Search Papers | Poster Sessions | All Posters

Contributed Talks II

Talk Session: Thursday, August 8, 2024, 9:30 – 10:30 am, Kresge Hall

9:30 am

Semantic action representations in the mind and brain

Diana C Dima¹ (), Jody C Culham¹, Yalda Mohsenzadeh¹; ¹Western University

Understanding others’ actions is an essential part of our everyday visual experience, yet the underlying computations are not well understood. Natural actions pose a complexity challenge, varying along many perceptual features. To address this, we annotated natural videos of everyday actions with a rich set of visual, social, and semantic features. In particular, we tested four models of action categorization defining actions at different levels of abstraction from specific (action verb) to broad (action target: an object, a person, or the self). We combined behavioral similarity judgments, EEG, and fMRI to investigate action representations in the mind and brain. Using variance partitioning, we found that the target of actions uniquely explained behavioral similarity judgments, as well as EEG patterns starting at 200 ms after video onset. EEG-fMRI fusion linked this processing stage to representations in the lateral occipitotemporal cortex. Together, our results show that actions are categorized primarily according to their target, and reveal the underlying spatiotemporal dynamics.

9:42 am

Deep neural networks reveal context-sensitive speech encoding in single neurons of human cortex

Shailee Jain¹ (), Rujul Gandhi¹, Matthew K. Leonard¹, Edward F. Chang¹; ¹University of California San Francisco

Speech perception relies on continuously tracking information at different temporal scales and integrating it with past context. While prior studies have established that the human superior temporal gyrus (STG) encodes many different speech features— from acoustic-phonetic content to pitch changes and word surpisal— we are yet to understand the neural mechanisms of contextual integration. Here we used deep neural networks to investigate context-sensitive speech representations in hundreds of single neurons in STG, recorded using Neuropixels probes. Through this, we established that STG neurons show a broad diversity of context-sensitivity, independent of the speech features they are tuned to. We then used population-level decoding to investigate the role of this property in tracking spectrotemporal information, and found that neurons sensitive to long contexts faithfully represented speech over timescales consistent with higher-order word and phrase-level information (~1sec). Our results suggest that heterogeneity in both context-sensitivity and speech feature tuning enable the human STG to track multiple, hierarchical levels of spoken language representations.

9:54 am

Pink noise in speakers' semantic synchrony dynamics predicts conversation quality

Kathryn O'Nell¹, Emily Finn¹; ¹Dartmouth College

Dyadic social interaction is a complex coordination task involving many interconnected variables. Previous research has shown that metastability -- persistence for an extended, but impermanent, period of time in a non-stable state of a system -- can be a useful lens for understanding what makes an interaction successful. Metastability occurs at certain noise signatures; namely, pink noise, in which the power of a signal is inverse to its frequency. However, this framework has thus far only been applied to para-conversational signals like heart rate and prosody -- not to the semantic content of a conversation. Here, we present pink noise analysis of semantic trajectories as a metric for conversational success and apply this technique to a large open conversation dataset. Our results demonstrate that adaptive movement in and out of semantic synchrony in a conversation predicts a host of variables representing conversation quality.

10:06 am

Decoding of Hierarchical Inference in the Human Brain during Speech Processing with Large Language Models

Joséphine Raugel¹ (), Valentin Wyart¹, Jean-Rémi King²; ¹Ecole normale supérieure, Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale, ²Ecole normale supérieure, Laboratoire des Systèmes Perceptifs, Centre National de la Recherche Scientifique

Many theories of language in the brain rely on the notion of predictions. Yet, little is known about how linguistic predictions effectively change the representations of language in the brain. Here, we investigate how two levels of representations in the language hierarchy vary with predictability: words and phonemes. For this, we rely on Large Language Models (LLMs) trained to predict incoming words and phonemes, and estimate the posterior probability of these features as speech unfolds. We then evaluate whether predictability impacts the representations of words and phonemes decoded from the MEG responses of 27 participants listening to two hours of stories. Our results show that both words and phonemes are best decoded from the brain if they are unexpected from a given context. This finding constrains the computational architecture underlying natural speech comprehension.

10:18 am

Modeling Multiplicity of Strategies in Free Recall with Neural Networks

Moufan Li¹, Kristopher T. Jensen², Qihong Lu^3,4, Qiong Zhang⁵, Marcelo G. Mattar¹; ¹Department of Psychology, New York University, ²Sainsbury Wellcome Centre, University College London, ³The Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, ⁴Center for Theoretical Neuroscience, Columbia University, ⁵Department of Psychology, Rutgers University

Humans preferentially recall items that are presented in close temporal proximity together -- a phenomenon known as the 'temporal contiguity effect'. In this study, we investigated how this phenomenon emerges naturally when training a recurrent neural network with episodic memory on free recall tasks, and the neural mechanisms underlying this process. The model managed to produce the temporal contiguity effect, and we found individual differences in neural mechanisms for different models. Some models learned an item index code that matches the `memory palace' technique and recalled in a forward order, while the other models learned to recall in a backward order and relied more on item-related temporal context. We found that the extent to which the model changes the context between encoding and recalling memories affects the learned recall strategy. Our findings provide insights into how different memory strategies may arise in human free recall.