Stephen Wilson 0:05 Welcome to Episode Three of the Language Neuroscience Podcast. I'm Stephen Wilson and I'm a neuroscientist at Vanderbilt University Medical Center in Nashville, Tennessee. This podcast is aimed at an audience of scientists and future scientists who are interested in the neural substrates of language. My guest today is Eddie Chang, who is Professor and Chair of Neurological Surgery at the University of California, San Francisco, UCSF. Eddie has been a friend of mine for a long time, from back when I was a postdoc at UCSF, and he was a resident. Eddie is a neurosurgeon who treats patients with epilepsy, brain tumors, and many other conditions. I've had the unique experience of being in the operating room and watching him do a complete brain surgery, including awake language mapping, seven hours start to finish, I learned many things from that, one of them being that if ever I, or any of my loved ones, need brain surgery, I know exactly who I want to do it. But Eddie is not just an amazing neurosurgeon. He's also an outstanding neuroscientist. He and his team have produced a brilliant series of papers over the last decade exploring the brain mechanisms of speech perception and production, primarily using the technique of electrocorticography, which offers unprecedented spatial and temporal resolution. Today, we're going to talk about just a few of his studies. Full disclosure, I have some past and ongoing collaborations with Eddie, but not related to the lines of work we're going to be discussing. Okay, let's get to it. Hey Eddie, how are you? Eddie Chang 1:31 Good. I'm great. Stephen Wilson 1:33 So we've known each other for about 12 or 13 years Eddie Chang 1:36 At least. Stephen Wilson 1:37 Do you remember the first place that we ever met up? Eddie Chang 1:39 I do. We were meeting at the cafe at UCSF across the street from the lab. Stephen Wilson 1:44 Yep. In the food court. Eddie Chang 1:46 Yeah. Stephen Wilson 1:46 It was seven, I actually looked it up in my email. It was like 7:30 in the morning. And we were discussing possible involvement of speech motor cortex in speech perception. Eddie Chang 1:56 That's right. I do remember that very clearly. And I remember how I felt, you know, quite excited to meet you. Because, number one, I was very excited about the work you had done before, but in particular, the discovery that you had using fMRI of auditory response in the motor cortex. So that was something we were very interested in that particular time. Stephen Wilson 2:19 Yeah, and I was pretty excited to see that you guys were observing the same area being involved with your, with your ECoG (electrocorticography) methodology. So anyway, I'm just gonna get started, I was wondering, like, you know, did you always want to be a neurosurgeon? Eddie Chang 2:34 No, I mean, to be honest with you, no, I don't think that that was part of my original programming. I would say that, I really didn't know actually, what I want to do until it really just happened upon me. Stephen Wilson 2:48 What were you interested in as a kid like in school? Were you, were you academically inclined? Eddie Chang 2:54 I would say modestly, I mean, I think the things that really interested me had to do with the natural world. I was very intrinsically interested in biology. At the same time, physical properties, and so, you know, I was drawn to the sciences, I think, pretty early on. Stephen Wilson 3:12 And then how did you end up in med school? Or I guess, like, did you start doing research before after you went to med school? Eddie Chang 3:21 Well before, during, and contining to, you know, all of the above. And I think in the beginning, it was more of a practical thing of just learning as part of learning and trying to understand scientific method and what it's like to do research. In the beginning, it was really more oriented around molecular biology and cellular biology. But what really got me excited, I think, was thinking about neurophysiology and in particular systems neuroscience. That got me very, very interested in literally hooked. Stephen Wilson 4:00 Uh huh. And it was auditory neuroscience that you got into first, right? Eddie Chang 4:06 Yeah, that's right. So when I was a first year medical student here at UCSF, I was really fortunate to sign up for a seminar that was taught by really, I think, a lot of luminaries in, in systems neuroscience, one of which was Mike Merzenich, and, you know, I was just really lucky to spend some time with him as a medical student, which I think was really a one off in terms of his, his teaching responsibilities. It just wasn't a regular part of the curriculum. But that year, there was a seminar series and after that, I decided, you know, I wanted to spend more time understanding and learning about neurophysiology. At that time, the thing that really drew me to this lab in particular was these ideas of cortical plasticity, and how you would measure that in the early experiments in adult cortical plasticity really, you know, really blew my mind. And I asked him if I could join his lab and and he accepted. And the model system at that time they were looking at was the developing auditory system. So that's that's how I first got my, my foot in the door. Stephen Wilson 5:20 Cool. And do you want to tell me and our listeners about that first big project that you did in his lab? Eddie Chang 5:28 Sure, it's interesting. As you know, science kind of goes in unpredictable ways. And so when I initially worked in his lab, you know, I proposed to do an experiment where we would essentially, in a controllable way, reorganize the auditory cortex so that the tonotopic organization could be essentially manipulated and reconfigured based on, you know, cortical plasticity mechanisms. And I'm still very interested in that topic. But it turned out to be not a great, not a particularly practical, or tractable problem, especially with the tools that I had at the time, especially as like a one year medical student experience. So I quickly changed gears. And I was really fortunate to be around other postdocs and neuroscience graduate students. And a lot of the focus at that time was on the developing auditory system, as I mentioned. The model system was the rodent. So we're very particularly interested in how plasticity is different between the infant brain and also compared to the adult brain. And, in particular, how the auditory system organizes around that. And at just the time that I joined Mike's lab, it had been demonstrated that there is indeed a sensitive or critical period for tonotopic reorganization of the auditory cortex in developing rodents. This happens, essentially, in the first month of life, postnatal life, for the infant rats. And we were really interested in what are the effects of certain kinds of experience on that development of plasticity. And the project, the very simple project that I had come up with in Mike's lab, was to look at the the effect of noise rearing on these baby rats. And the reason why we were thinking about that was that we're looking for some analog to visual cortex experiments where, you know, there's various forms of sensory deprivation, like dark rearing, or monocular deprivation, basically keeping one eyelid shut. And there's no clear analog, because if you, there's not a good way to reversibly, reverse, you know, reversibly limit the hearing in one ear or the other or both. So we went the opposite way, which was to try to saturate the hearing, so that instead of depriving the infant rodents from hearing any sounds, we saturated, increased the noise level, thereby decreasing the signal to noise. So what they were really deprived of was not sound, but patterned sensory inputs in sound. And what we found was something quite dramatic, which was that when you had environments that were saturated with noise, that the auditory cortex development was basically on hold. It was paused indefinitely, while the animal, you know, didn't have access to patterned sound stimuli. So the takeaway from that for us was that it is really quite important that it sound input have important role in establishment of the organization in auditory cortex. Stephen Wilson 9:19 Did the animals, after they came out of the period of noise rearing, were they able to sort of reopen a critical period afterwards? Or was it kind of like, it was done if they had missed out on it? Eddie Chang 9:29 No. So that that's a great question. And one of the main findings from this paper, which has been, I think, replicated from several labs now, which is that in animals that were exposed to normal environmental sounds, the critical period essentially was closed within that first month of life. But if the animals were deprived of that sensory input, with saturating noise, that you could keep it open indefinitely. So that we saw that there were these very inducible forms of plasticity, just by animals listening to pure tones, for example, that could induce plasticity months later, far beyond what we thought was the critical period. Stephen Wilson 10:14 Wow, that's exciting. It sort of suggests that maybe the reopening of plasticity might be more accessible than we generally think, in other contexts too maybe. Eddie Chang 10:23 Yeah, it's possible. In those experiments, it's really hard to say that it was a reopening of the critical period. It's more like it never really closed. Stephen Wilson 10:33 All right. Yeah. Okay. I guess that's kind of a different situation. Eddie Chang 10:36 Yeah, it was, like indefinitely open until it got really the sound inputs. And so, you know, in all honesty, I kind of lucked out, because I happened to be at the right place at the right time when there was a lot of interest in developing auditory system. And it was definitely a simple experiment that, you know, where we found something quite analogous and powerful, in the auditory system that been shown in vision. So that really, really introduced me a lot into the methods and thinking about the neurophysiology of the auditory cortex. And that's never changed since then in reality. Stephen Wilson 11:19 Absolutely. Eddie Chang 11:20 Yeah. So back then I was introduced to the ideas of really thinking about sensory representation. And this particular question of how speech sounds might be represented, through development and through cortical mechanisms. And I think the rest is history in the sense that I've only tried to find the answers to that more directly over time. Stephen Wilson 11:45 Yeah, I bet you didn't expect me to ask you about that old auditory neurophysiology paper. Eddie Chang 11:51 No, I didn't. No. Stephen Wilson 11:54 So yeah, so you became a speech scientist. Along with several other things, obviously. How did you make that transition into your interest in speech perception, that's basically been your focus for the last decade? Eddie Chang 12:08 Well, I think that, you know, I don't have formal training in speech science, as you know. Technically, I don't have formal training in neuroscience either. My formal training is really in medicine. But I've had a lot of informal indirect research experiences, and informal, indirect educational experiences. But the most important thing is being able to work with a wonderful set of collaborators who I've learned from tremendously. But also, I love this topic. It's the kind of thing that keeps me up really late at night reading in bed. And a lot of it is just self learning and being inspired by what others have done beforehand. Stephen Wilson 12:53 Cool. So can we talk about your 2010 paper, which is one of my favorite one, one of my favorite of your papers? This is an ECoG paper. And I don't think that everybody that's going to listen to our show, is going to be familiar with that methodology. So I was thinking, maybe we could kind of start with talking about, like, what ECoG is, you know, under what circumstances can you do it? What are the strengths and weaknesses of it as a methodology? And then we can kind of get into the paper after we do that groundwork? Unknown Speaker 13:23 Okay well, let me let me put some context. So first of all, ECoG is shorthand for electrocorticography. That's a mouthful, but basically, what that refers to is a method of neurophysiological recordings, where an electrode is placed on the brain surface, not penetrating into the brain, but on the brain surface, and passively records, the electrical field potentials from around, you know, I would say tens of thousands of neurons simultaneously. And it's a method that you can record directly from the brain with very high spatial and temporal resolution, because the electrodes are usually assembled in an array. And so you can record from multiple sites simultaneously, that, you know, can be as low as just, you know, millimeters apart, with the electrophysiological recordings happening on the timescale of milliseconds. Stephen Wilson 14:23 So you've typically got an array of, let's say, I think back in these days, at least, it was like an eight by eight array of electrodes, roughly? Eddie Chang 14:30 Yeah. So typically, when we started with a lot of this, the clinical recordings that we were doing in the hospital for for treatment of patients with epilepsy was on the order of eight by eight electrodes, 64 channels, with one centimeter spacing. And the natural question, of course, was, what would this physiology look like if we increase the spatial resolution, and look at a much finer scale and so on. I spent a lot of my time early on actually trying to just figure out how we could do that, do higher resolution recordings, both in the operating room during awake surgeries, but also implanted in the epilepsy monitoring unit, in a way that was ethical and consistent with our care goals, but also piggyback some neuroscience experiments. Stephen Wilson 15:26 And did you, were you able to increase the resolution? Eddie Chang 15:29 Oh, yeah. So the first thing that we did was we went from the eight by eight to a 16 by 16 channel. So within that first year, setting up our research group, we had 256 channel recordings standardly as part of our recordings. And, again, we did it in a way that in no way compromised care, and in some cases, enhanced it, but allowed us to address questions at a much finer scale. And, you know, we really just didn't know what direction this would go. I remember at the time, with the one centimeter spacing, kind of physiology we were doing, it had great temporal resolution, but the spatial was not great. And so the type of questions were very different, it was sort of like looking across broad swaths of cortical areas and asking, you know, when does auditory cortex come online compared to let's say, frontal cortex or parietal, and once we were able to do higher resolution, the nature of the questions started to change very dramatically, you know. It's not about this inter aerial timing of activation, but really more about what's happening locally, within a given gyrus, for example, and, you know, that's been our focus, actually, for the last decade, is trying to understand that resolution, on the order of millimeters in the superior temporal gyrus in particular. Stephen Wilson 16:58 Yeah. So just to kind of recap, you've got, you know, you're making direct recordings from electrodes placed on the brain surface, during surgery, or during presurgical monitoring after the skull has been removed for upcoming surgery, right? Eddie Chang 17:15 Right. Stephen Wilson 17:15 And you've got a spacing now of like, probably five millimeters between electrodes now that you've got this kind of larger grid or this? Eddie Chang 17:25 Yeah, it's usually three or four millimeters. Stephen Wilson 17:27 Okay, so three or four millimeters between the electrodes, and you've got essentially, like all the temporal resolution you could want, I mean, nobody's gonna want to go beyond a millisecond, I think, although I'm probably just being narrow minded. So in this 2010 paper, you looked at how the areas under the electrode grid, which almost always involved temporal cortex, as well as frontal and parietal, how they would respond to a series of syllables that were stepping between ba, da, and ga, right? Eddie Chang 18:01 That's right. Stephen Wilson 18:02 Can you tell us about the, you know, I think most of our listeners probably are familiar with the whole notion of categorical perception, but maybe it's a good thing, just to kind of like, lay out the question that you were addressing there. Eddie Chang 18:16 Okay. Yeah. So the bigger question around this is, what is the nature of the cortical representation of speech sounds? Okay. So we're interested in how basically, neural activity in the auditory cortex, the higher order auditory cortex that we call the superior temporal gyrus, how that corresponds to the sounds of speech. And historically, there's been this phenomenon that was really popularized and made quite important by Al Lieberman that basically showed that the auditory system... Behaviorally, you know, they showed that basically, people are not really sensitive to some of the fine details of the acoustic information. And then we're very sensitive to the phonetic information. And the implication was that it's some part of the brain that processes speech, that the nature of its computation, and its representation, is the level of phonemes, abstract units that give rise to speech, as opposed to the what we call the veridical or the parameterized details of the sounds themselves. So that's generally what we call categorical perception, which is you lose some of the variance from the acoustic stimulus in support of a mental representation, internal one, that is organized around this idea of phonemes. So the basic experiment that he (Al Lieberman) did, which was to take a set of synthesized consonant-vowel tokens that varied in the onset position of the second formant, and what he showed was basically that if you vary the second formant in these these consonant-vowel syllable tokens, you can create a continuum of sounds from ba to da to ga, just by changing that F2 parameter in a very linear and parameterized way. But the behavioral effects of it were not linear. They were categorical in the sense that people did not perceive a continuum of nine or ten different tokens, but in fact, three discrete speech sounds. Stephen Wilson 20:37 Ba, da, and ga. Eddie Chang 20:39 That's right. Ba, da, and ga. Stephen Wilson 20:41 And I think I think I have your stimuli that you've sent me at some point in the last 12 years. So I'm probably, if you don't mind, I'm just going to drop them in the recording here for our listeners. I mean, I'm not going to do it now. But I will just, yeah play your ba da ga stimuli. Eddie Chang 20:54 Sure. I mean, it's probably much easier just to experience that. And for me to describe it anyways. Example stimuli 21:04 [continuum of CV syllables] Eddie Chang 21:08 So the basic question is, in the superior temporal gyrus, the high order auditory cortex, part of what we know of as Wernicke's area, how are these sounds processed? Is the neural representation closer to the sound properties? Or is it closer to what our perception is like in terms of these discrete and categorical phonemes. And... Stephen Wilson 21:36 Right, it's a really good test case, because the series of sounds, like there's like a dozen sounds, and they're all kind of, they're all, from an auditory, from a purely auditory point of view, they're linearly, they're equally spaced from one another, you've just kind of got these 12 sounds on a continuum. But from a perceptual point of view, they just group into these three phonemic categories. And so you can really ask that question like is, is the lateral surface of the superior temporal gyrus, representing the stimulus in a veridical auditory manner, or in a categorical linguistic manner, right? Eddie Chang 22:08 Yeah, that's right. And, you know, at least I think that it's processed in a way that it that really is quite discrete in the sense that you don't get three phonemes that you feel like are part of a continuum, they just really feel like three different, three different speech sounds, that don't really have a physical relationship to each other. That's certainly not intuitive. Stephen Wilson 22:33 Yeah. So what did you find? Eddie Chang 22:37 Well, what we found was that when we looked, you know, at this high resolution from the superior temporal gyrus, that if you looked at the population, that is, you know, among the dozens of electrodes that were activated by sounds, that the pattern of activity was organized in a way that supported a representation of, that was categorical, that it was supporting a discrete representation for the ba, da and ga sounds independently. And, in fact, something that was really, I think, consistent with the perceptual effect, which was that there was very little relationship, you know, at the population level between what we were seeing being coded and what was linearly parameterised, in the stimulus design. Meaning that we had a readout that was really like the ba da ga sounds as opposed to the 12 tokens that were varying in their F2 formant. Stephen Wilson 23:38 Yeah, that's really cool. Eddie Chang 23:40 Yeah, I mean, it was, it was great to see, I think it was, you know, there were a couple hypotheses that we had about how that might play out. And that work was a result of my postdoc with with Bob Knight. So during my residency in neurosurgery, I spent a year with Bob out at Berkeley. And I have to give Bob a lot of credit in the sense that not only was he an amazing mentor, and still is, but, you know, I happened to join his lab at a time when he was really pioneering a lot of the intracranial recordings from collaborations with some of my clinical mentors at UCSF like Nick Barbero and Mitch Berger and others. And he, I think, very early on recognized the potential of this and so I spent that year with Bob, with this as my research year project. Stephen Wilson 24:40 What a great project. Okay, so after that project you kept on, you continued on this line of work. And the next paper I was hoping we could talk about is a paper that's led by one of your postdocs Nima Mesgarani from 2014. I remember when you showed me this data at a conference. It was probably Neurobiology of Language because that's basically the only conference that I go to And I remember you pulling me aside and showing me this data and you erre excited about it. And I was excited about it too. So can you tell us about that study and how it kind of like goes beyond the, the work we've been talking about so far? Eddie Chang 25:14 Sure. I'm happy to tell you about that story. But I think in this context, as you know, Stephen, behind every paper and scientific discovery, there's a story of the people. So I just want to briefly mentioned how I first got in touch with Nima as a collaborator, as a postdoc for my lab. And yeah, again, I was very much, you know, I grew up in the auditory cortex neurophysiology world. There were several years that I was not involved in it, primarily because I was learning how to become a surgeon, a neurosurgeon, and, you know, really had no research going on whatsoever. But once, you know, I was in Bob's lab and reconnecting and thinking about the opportunities, and with that earlier experiment that we were just talking about proving to myself that there's something that could be done with high resolution ECoG, as opposed to, let's say, single unit recordings, which is what I think, Plan A was, to eventually do. You know, that was the goal. Definitely had been shown before that you can do this. But I was, you know, I was very attracted to the fact that this was safe and easy to do, and could be done quite easily. And so once we established that there was something to see at this high resolution, you know, we didn't turn back. And I remember early on in this that I was talking to a friend of mine actually who was in Bob's lab, that he really didn't think that we would be able to resolve anything of relevance to phonetic detail. And we'd only really be asking questions on areas of localization, but not really, questions that are more resolved, like what's the nature of the sound representation as it relates to specific features, in the auditory encoding. So again, this was all empirical, we just didn't know. But with that first experiment, it gave us confidence to move forward and keep thinking about this. And so when I was re-engaging with the auditory neuroscience community, I was at one of these SFN meetings at a satellite. And I met Nima at one of the post parties. And in some ways, it was love at first sight, in the sense that, you know, I encounter someone who first of all was very, very nice and easy to talk with. But secondly, had extremely good training in electrical engineering, and speech sciences, and auditory neuroscience with Shihab Shamma. And at that time, he was doing a postdoc at Hopkins in electrical engineering and speech sciences. And I told him what I was trying to do in setting up a new lab. At this time, I hadn't even finished residency, but was teeing things up for starting the lab. And you know, we spent that whole time at that party just really brainstorming and thinking about what could be done, and where would we start. And we started collaborating, initially at a distance, to the point where we were collecting some data before he even showed up in San Francisco. Stephen Wilson 28:45 And he had been I don't know, had he already, he was presenting speech sounds to rabbits, right? Eddie Chang 28:51 Um, close, yeah. Stephen Wilson 28:52 Was he already doing that before at this time or before? Eddie Chang 28:54 Right. In what, what, that's exactly. That's very close. Basically, Nima had been working with Shihab Shamma where they were doing recordings from the ferret auditory cortex. Stephen Wilson 29:10 Oh, ferrets, yeah, I'm sorry. Eddie Chang 29:12 Ferrets. And looking at single neuron responses to speech sounds and in many ways the experiment they did in ferrets was exactly the first one I wanted to start with in humans, where you would take a number of natural English sentences that were phonetically transcribed and, in a broad way, just characterize the tuning across the auditory cortex to different speech sounds. Stephen Wilson 29:41 Yeah, well, I'm embarrassed that I got my little furry creatures mixed up. But yeah, so... Eddie Chang 29:47 Yeah, so I think it was like for me, again, very fortunate that I encountered Nima. We had an amazing partnership. When he came out to join,I do have to say that in reflection, it was really high risk, because we're talking about a new methodology and clinical patients. But it turned out to be an amazing partnership in the sense that there was so much to discover, actually, with this kind of data and with a lot of open ended questions. So just to be clear, which paper are you talking about? Stephen Wilson 30:25 The 2014 paper with the, showing the similarities of organization between phonemes in the STG (superior temporal gyrus) and phonemes according to linguistic theory. Eddie Chang 30:39 Right, okay. So that was a project that I think was kind of obvious to do. It was very obvious as a next step to play natural speech. And to use what we call encoding models, basically, models where you're trying to predict the neural responses and the variance in the neural responses, as it relates to specific attributes of the stimulus. In this particular case, we're interested in the spectro-temporal properties of speech and also the linguistic phonetic properties, what we call phonetic features. And those phonetic features generally refer to articulatory properties, like properties by which the sounds are made, as well as unique acoustic properties that we call things like fricatives, or plosives, etc. These are just different features that relate to how the sounds are produced. And, you know, again, relying on this really amazing corpus that was developed at MIT called TIMIT, which we still continue to use, already transcribed for phonemes, we could essentially play this to the participants in these studies, and take the data back to the lab, and look at what was happening in the superior temporal gyrus as people were hearing these sounds and encoding this information in the brain. Stephen Wilson 32:19 Yep. Okay. So when you looked at the representations in the brain of the different, so you're extracting all of these, each... So the neural response to each phoneme is being extracted from the average of responses to it in many different phonetic contexts averaged across the set of naturalistic sentences, right? So when you looked at the neural representations of these phonemes, what kind of patterns did you guys see there? Eddie Chang 32:47 Yeah. So just to put in other words, like every time, you know, there was an [a] sound, we could look at the average responses across the electrodes, every time there was a [b] sound, we could look at, you know, which electrodes were sensitive to that, and you're right, it was really looking at the, the average and outside of the context of the, of the words in sentences. And what we found was that, if you looked at the individual electrodes, the tuning was to these individual phonetic features. Not phonemes themselves, not the discrete units. Like, you know, [a], [b], the different vowels and different consonants. But in fact, the subunits of those phonemes that we call the acoustic phonetic features. Stephen Wilson 33:04 Right. Yeah, I remember that the paper has a very striking figure in it, where, you know, you can see this one particular electrode, I think it's the first one, it's probably called like e1, or something like that, or a1, it's like, it's just got some arbitrary label. And you kind of show how it responds to all the different phonemes of English and it's like, [p] [t] [k] [b] [d] [g], it goes crazy for all of those. And then it doesn't respond to any other phoneme at all. And like, if you're not a linguist, you'd be like, Oh, well, whatever, it responds to like six random phonemes, but if you're a linguist, or, you know, if you know something about phonology, you'd be like, okay, it responds exactly, to all the stops, regardless of whether they voiced or voiceless, and it responds to no other sounds. And so it kind of looks like it's representing this linguistic feature. It's like very principled, the set of sounds that it's responding to. Eddie Chang 34:38 Yeah, that's exactly right. One of our hypotheses going in was that we were going to see tuning to individual phonemes. But in fact, what we found was that individual electrodes were primarily tuned to sets of phonemes, classes of phonemes that shared a property in this particular case, it was ones there were different properties like plosive consonants, or fricatives, or low back vowels, high fronted vowels, nasals. And it was striking to just see that we could see this not on the order of just one electrode at a time, but dozens and dozens, across multiple participants. And essentially seeing that there was this dictionary of phonetic feature tuning for every acoustic phonetic feature, but in part, also giving rise to every potential phoneme in English. Stephen Wilson 35:41 Yeah, that's really cool. And that's kind of how I mean, I think linguists don't really think that phonemes are discrete entities either, right. I mean, a phoneme is really nothing more than a bundle of features. Because if you kind of get into autosegmental phonology, then you know, that's all phonemes are, and like a phonological rule is just what happens when a feature starts to spread out across adjacent segments, and so on. So, you know, as you mentioned, like a lot of these phonological features are sort of articulatory in nature. But yet they also sound... like phonemes that have similar features also sound similar, or they have similarities in their acoustic realization. So to what extent do you think the patterns that you observe are driven by the acoustic nature of the stimulus only, as opposed to kind of its linguistic representation? Do you have ways of addressing that question? Eddie Chang 36:31 Yeah, I mean, I think that that basic question that you just posed, right? Is it more acoustic or linguistic? has been at the heart of nearly all of the work that we've done in the auditory cortex. And I've come to the realization that in some ways, it's not that easy to answer definitively, possibly, because it's even ill posed. So the way that we think about it now is that these representations are intrinsically auditory in nature. But they're not simple low level kinds of auditory encoidng They're, in fact, high level forms of auditory encoding, things that you would not necessarily see from lower parts of the auditory system. What I mean by that is, you know, these kind of response to, to, that are normalized by speaker identity, that are much more sensitive to contextual aspects, meaning like changes in an acoustic parameter, as opposed to the absolute acoustic parameter itself. So an example is pitch, where we see tuning to relative pitch and pitch change. And that is far more corresponding to the prosodic changes that we have in intonation than absolute pitch, which is a useful queue for things like speaker identity. And so the nature of the encoding that we think about a lot is something that's higher order auditory, but interfaces really importantly, with linguistic and phonological representations. And the way that that's realized, I think, is not only locally at the small populations that we record from, but more importantly, from the population, the global population, what we call the ensemble of all the recorded responses across all of these different sites. And that is what we refer to as sort of like the emergent representation of phonological structure, which then goes beyond sort of like a simple acoustic representation. Stephen Wilson 38:51 Yeah. I feel like it's an answer that's not a non-answer. Like, I feel like it's maybe it is well posed. And the answer is that it's a pretty linguistic representation. Like, you know, given all of the lines of evidence that you just mentioned, you know, that it's not kind of responding to simple auditory properties, but it's responding to a sort of derived auditory properties that are linguistically relevant, and that it probably, at least in part, language specific, I mean, like that might even differ across different languages. Eddie Chang 39:24 I guess what I mean by ill posed is that, you know, you kind of present it is like this "or" situation, right? It's auditory "or" linguistic. And what I'm saying is that, at least the way we think about things right now, it really is at the interface of those two. And the implications for that are that you might imagine, for example, that if there's an "or" situation that you have to passage between a part of the brain that is, quote unquote, doing auditory processing to a different part of the brain that is doing linguistic processing. And I don't think that that is what we see or what I'm describing or where we're converging. I think what we're talking about is a part of the auditory system that is very high order. But the nature of its tuning is not specific to speech at a local level, for sure. But its processing is specialized for speech in the sense that what it's actually tuned to, and the statistics of speech, are very much well represented there in a way that it is, it's much better, you know, for speech compared to other natural sounds. So, that's where we're kind of coming down and converging in our thinking about this question of acoustic versus linguistic. Stephen Wilson 40:47 Okay, yeah, I see what you're saying. Yeah, great. Eddie Chang 40:50 You know, so basically, 2014, you know, and beyond, with Nima's paper, we had essentially shown these properties with English using TIMIT, and I think for several years after that, we continued to look at a lot of these features in English. And at one point, in English, we're looking at prosodic aspects of English, in particular, the intonational stuff that's related to pitch change, when we stress particular words is lexical stress, and prosody. And, as you know, in English, you know, pitches primarily used for that kind of that form of communication. But in other languages, what we call tonal languages, such as Mandarin, several Asian and African languages, that pitch information is actually used as a cue in a really different way. It actually has lexical level information, like word level. It's treated much, much different linguistically than it is, in many ways, it operates more like a phonetic cue, that can directly link to different lexical meanings. Stephen Wilson 42:09 Just as a brief, personal interlude, you're a native speaker of a tonal language, right? A native bilingual? Eddie Chang 42:17 I think I was native. And then it got completely wiped out. By my upbringing, in California. So I would say I was native and lost it, and it got really replaced by English as needed. My parents were immigrants to the US, and definitely Mandarin native. That's what they spoke to me early on in life. But like I said, you know, I was born here and really grew up in an English environment. So I do consider it my native language. But actually, that kind of leads to this question. I mean, that there was a natural, intrinsic question. And I was very personally interested in this question about Mandarin versus English. So, you know, fast forward from the studies with TIMIT in 2014, with Nima to, you know, to three years ago, where I started thinking about how to broaden our work beyond English. And the reason why I was interested in that was of these really longstanding questions about, you know, what, if there's such thing as like, a universal phonetic inventory, a universal code, you know, for for speech processing. And I encountered one of my colleagues who's a neurosurgeon in Shanghai, his name is Dr. Jinsong Wu, who has become quite a, you know, quite a leader actually in brain tumor surgery in Shanghai, as well as well as a researcher. And we met at one of these neurosurgery meetings, and he was asking me about some of our work. And we said, you know, wouldn't it be cool if we could compare neural responses in people who are native Mandarin speakers, with those who are native English speakers. And we thought, in order to do this in the best way, we should do a controlled experiment where these native listeners listen to, not only their native language, but also the other language. So we had, we developed an experiment basically where the native Mandarin speakers in Shanghai would listen to Mandarin and English. And our English speakers here in San Francisco, were listening to both English and Mandarin. And with that design, we focused and started this experiment on the representation of pitch. Because on the one hand, in English speakers, you've got to use of pitch as primarily used for prosodic information like we've talked about. And in the Mandarin speakers prosodic and lexical level information has been conveyed. Both use pitch in really important ways, but in different ways. And so it was a great, you know, like I said, there's a story behind all of this. And Dr. Wu actually sent one of his trainees who spent a year with us in the lab, Dr. Junfeng Lu. And I sent my graduate student there, Claire Tang, who spent a couple of months in Shanghai, where we were doing this cross exchange of people. And a new postdoc joined the lab from CMU, his name was Yuanning Li, who is bilingual, truly bilingual in English and Mandarin, who took on this project. And so what we were interested in understanding is, is really just how the processing of an acoustic parameter, pitch, how it's processed in these different linguistic contexts. So if your native language in and non-native, and basically what we found was that in both Mandarin speakers and English speakers, that when you looked at individual electrodes, you do see this really clear cut code for pitch change and relative pitch. So what I mean by that is that when the English speakers were listening to English, we could essentially find these electrodes that were coding for pitch change and relative pitch. But when they were listening to Mandarin, these electrodes that were tuned to these properties, also were active, even though it was like a non-native language. And we saw the exact same thing in the people who were studied in China. So that first result is that when you look at single electrodes, you see this kind of language-invariant tuning to things like pitch change and pitch. So the second question really related to well, if that's the case, how does this language-specific sensitivity arise? And where does that come from? How's it processed? What's its mechanism? And the answer to that story was when we started looking beyond individual electrodes, where we're looking at the distribution, and sensitivities of not just one electrode, but again, dozens of electrodes, and their tuning to these parameters of pitch change and relative pitch. And when you look at the population, and you analyze it as a population, then you actually start to see the sensitivity. And what we basically found was that in the Mandarin, native Mandarin speakers, that when you looked at the population level, that there was sensitivity actually around tone categories. Stephen Wilson 48:23 Uh huh, which were not there in the English speakers, huh? Eddie Chang 48:26 Right. That's right. When you look at the population in English speakers, it was sort of like this smooth continuous distribution of pitch values. But in the Mandarin speakers, we have found, in fact found, again, at the population level, a categorical effect or a warping effect, at least. Stephen Wilson 48:46 Right. And none of this would be observable at the level of individual electrodes, huh? Eddie Chang 48:50 That's right. Stephen Wilson 48:51 That's really fascinating. I think a lot of people are going to very much like that result. Eddie Chang 48:55 Yeah, I mean, I think that, you know, it might be an intuitive result. We definitely had a lot of different hypotheses going into how this might happen. You can imagine that there's different parts of the brain that are encoding these different native sounds. You can imagine that there would be poor tuning from one language to another. But what we actually found was when you looked at the individual electrodes, and you, let's say built a model that was based on the stimulus response for these changes in English, if you built this in English, on the data, from people listening English, it in fact, did quite well when people were listening to Mandarin. It actually did almost equally well, at the local level. So then it really comes down to, what's the distribution of all of these different electrodes that are tuned to different aspects of pitch? Some were tuned to pitch going up, some were tuned to pitch going down. And we found that the relative distribution across Mandarin and English speakers was different. So as an example, in Mandarin speakers, we found more electrodes that were tuned to downward trajectories of pitch that in English are there but may not have the same linguistic significance, for example. Whereas in Mandarin, the fourth tone category heavily relies on that downward pitch trajectory. So, I think this alludes to a theme that I brought up earlier, which is that, again, when you look locally, what you see is something that is quite tuned to acoustic parameters. I call them high level, because this is not the absolute pitch. It's the derivative of the pitch, and it's contextual, relative pitch, meaning that it's a normalized pitch signal, meaning that different speakers have different pitch dynamic ranges. But these neural populations tune out that kind of pitch information about whose whose mouth is it coming from, and are very tuned into the dynamic properties, which is the change in the pitch that we use to convey information about prosody and lexical information. And that is not language specific at all, apparently, at this local level. But when you look at the population, and look at distribution of all this together, then we see the emergence of things that are tuned to a given language. And I think at that level really, is at the level that we think things are happening at a phonological level. Stephen Wilson 51:38 Oh that's just fascinating. Is it a preprint yet, or is it not quite there yet? I mean where are you at with that project? Eddie Chang 51:43 Oh yeah, I mean, we probably should post it as a preprint. But I assume maybe it will be out in the next couple of weeks. But look out for it. We're really excited about it. It's like, a really kind of dream experiment, to be able to do this trans-Pacific kind of experiment. And, you know, we just got a grant to look at this in bilinguals, as well as English, Spanish. So it's an area that I think is very fascinating, actually. And a lot more to do. Stephen Wilson 52:18 Yeah, no, that's extremely cool. Okay, so we've talked quite a bit about your work mostly around speech perception, I had sort of planned to talk to you about speech production as well, but I just don't think it would do it justice, if we had tried to cram all that in. And there's a whole nother topic, I would like to talk to you about some time, which I think people would find really interesting, which is the experience of being a surgeon and you know, operating around language areas, and you know, under what circumstances do aphasias result and so on and so forth. So I think that there's like whole other strands of your work that I think I'd love to talk to you about again, some other time, if you could, you know, maybe down the road, chat about these other things to? Eddie Chang 53:00 Yeah, happy to, any time. Stephen Wilson 53:03 All right, cool. Well, thanks a lot for your time, and I'll catch up with you again soon. Eddie Chang 53:09 Thanks, Stephen. Good seeing you today. Stephen Wilson 53:11 Yeah, good to see you, too. Okay, well, that's it for Episode Three. Please subscribe to the podcast on your favorite podcast app. And if you have time, rate and review the show on Apple podcasts. If you'd like to learn more about Eddie's work, I've put some relevant links and notes on the podcast website, which is langneurosci.org/podcast. I'd be grateful for any feedback. You can reach me at smwilsonau@gmail.com or smwilsonau on Twitter. Okay, bye for now.