Search Papers | Poster Sessions | All Posters
Poster C55 in Poster Session C - Friday, August 9, 2024, 11:15 am – 1:15 pm, Johnson Ice Rink
Brain-like functional organization in Topographic Transformer models of language processing
Taha Osama Binhuraib1, Greta Tuckute2, Nicholas Blauch3 (); 1Novus Technologies, 2Massachusetts Institute of Technology, 3Harvard University
Topographic organization is a key feature of biological brains. However, representations within most machine learning models lack spatial biases, instead manifesting as disorganized vector spaces that are difficult to visualize and interpret. Here, we make two contributions. First, we introduce a new family of spatially-constrained topographic Transformer ("Topoformer") models. We train a 16-layer Topoformer model on a masked language modeling objective, and demonstrate significant topography in the learned responses. Second, we investigate an fMRI dataset of sentence-level responses to 1,000 sentences and demonstrate that human fronto-temporal language-responsive areas exhibit topographic response variability, variability which shows significant alignment with that of the Topoformer model. Our results motivate further examination of functional topography of language representations in brains and models, along with a task-optimized approach to topographic modeling more generally.
Keywords: Transformer Topography Neuroscience Language