Search Papers | Poster Sessions | All Posters
Poster C59 in Poster Session C - Friday, August 9, 2024, 11:15 am – 1:15 pm, Johnson Ice Rink
Human predictions of spoken language are more accurate and better aligned with neural activity than language model predictions
Thomas Botch1 (), Emily Finn; 1Dartmouth College
Humans communicate through both spoken and written language, often switching between these modalities depending on their goals. We investigated the alignment of large language models (LLMs) and human participants (N=300) that predicted words within a story presented as either spoken language or written text. We found that LLM predictions were more similar to humans' predictions of written text, though humans' predictions of spoken language were the most accurate. By training encoding models to predict neural activity recorded with fMRI to the same auditory story, we showed that models based on human predictions of spoken language better aligned with observed brain activity compared to models based on either LLM predictions or human predictions of written text. These findings suggest that the structure of spoken language carries additional information relevant to human behavior and neural representations.
Keywords: language LLMs alignment fMRI