Search Papers | Poster Sessions | All Posters
Poster A152 in Poster Session A - Tuesday, August 6, 2024, 4:15 – 6:15 pm, Johnson Ice Rink
Mapping Multimodal Conceptual Representations within the Lexical-Semantic Brain System
Emily X. Meschke1, Jack L. Gallant1 (); 1University of California, Berkeley
The representation of modal (e.g., visual, auditory, motor) and multimodal conceptual information underlies people’s ability to reason and communicate about the world. However, little is known about the form and spatial organization of multimodal conceptual representations. Some studies suggest that lexical-semantic representations, located within the broadly distributed lexical-semantic brain system, play an important role in linking conceptual representations across modalities. We therefore consider whether regions within the lexical-semantic system represent the same semantic information during narrative speech comprehension and movie watching. Eleven participants listened to narrative stories and watched compilations of silent short movie clips while their brain responses were recorded with fMRI. Two separate voxelwise encoding models were created for each participant: one to map lexical-semantic representations during narrative comprehension, and one to map visual-semantic representations during movie watching. Multimodal regions within the lexical-semantic system were then identified as regions where both models predict responses across the two experiments. This cross-experiment prediction analysis revealed a set of multimodal convergence zones that border the lexical- and visual-semantic systems, as well as a set of multimodal patches distributed within the larger lexical-semantic system. These results support the hypothesis that semantic representations within the lexical-semantic system are involved in linking conceptual representations across modalities, and in forming our conceptual understanding of the world.
Keywords: voxelwise encoding models multimodal representations semantic cognition language