Search Papers | Poster Sessions | All Posters
Poster A90 in Poster Session A - Tuesday, August 6, 2024, 4:15 – 6:15 pm, Johnson Ice Rink
Word-Order Error Detection Helps Data-Efficient Language Models Learn Syntax
Alexander Fung1, Chengxu Zhuang1, Steven Piantadosi2, Jacob Andreas1, Evelina Fedorenko1; 1MIT, 2UC Berkeley
Neural language models (LMs) require vast amounts of data to master syntax—a set of rules for how word arrangements create complex meanings. In contrast, children learn efficiently from a small amount of linguistic input. Inspired by findings of early sensitivity to word order information in children, we here augment LM training with a novel objective that emphasizes word order, in an attempt to minimize the data efficiency gap between LMs and humans. The new objective requires discriminating between grammatical sentences and sentences with word-order perturbations. After training LMs on developmentally plausible amounts of data, we find that LMs with this augmented training outperform control LMs (trained on standard masked language modeling) on select components of an established benchmark of syntactic knowledge (BLiMP) and a new benchmark we developed that targets word-order error detection. These results suggest that integrating synthetic tasks can effectively reduce the data efficiency gap between neural LMs and human learners.
Keywords: syntax learning neural language models word order masked language modeling