Search Papers | Poster Sessions | All Posters
Poster B68 in Poster Session B - Thursday, August 8, 2024, 1:30 – 3:30 pm, Johnson Ice Rink
Inferotemporal Cortex Underlies Primate Generalization Capabilities and Brain-Aligned Models Generalize Better
Marliawaty I Gusti Bagus1,2, Ernesto Bocini1, Tiago Marques2, Sachi Sanghavi2, James DiCarlo2, Martin Schrimpf1,2 (); 1EPFL, 2MIT
Primate inferotemporal cortex (IT) has been linked to the remarkable human ability of visual object recognition. The linear linkage hypothesis posits that a linear readout of IT neural activity predicts human behavior in core object recognition tasks within the domain of naturalistic images. We here ask whether this hypothesis explains human ability to generalize across image distributions. Specifically, we test if the representations encoded in primate IT combined with a fixed linear readout are sufficient to recognize objects across a variety of image styles such as cartoons, paintings, and sketches. We find that a linear decoder trained on primate IT responses to one image style is -- without any additional fitting -- able to classify IT responses to other image styles. The predicted performance of such a decoder, with a plausible number of neural sites and naturalistic stimulus training, corresponds to human accuracies across test domains. In artificial neural network models, we find that the more similar models are to primate IT, the better they generalize. When explicitly training models for IT alignment, generalization accuracy increases in correspondence with increased IT alignment. Our findings support that representations encoded in primate IT enable generalization to novel image distributions with a fixed linear decoder.
Keywords: Deep neural networks Object Recognition Primate Ventral Stream Out-of-distribution generalization