Search Papers | Poster Sessions | All Posters
Poster A103 in Poster Session A - Tuesday, August 6, 2024, 4:15 – 6:15 pm, Johnson Ice Rink
Convolutional neural networks align early in training with neural representations
H.Steven Scholte1 (), Julio Smidi1, Jessica Loke1, Niklas Müller1, Iris Groen1, Marcel van Gerven2; 1Universiteit van Amsterdam, 2Radboud University Nijmegen
Task-optimized deep convolutional neural networks (DCNNs) achieve human-level performance in object recognition and are leading in explaining neural activity across various brain measurement modalities. DCNNs are trained over numerous iterations to improve performance on a task, typically object recognition, whereby the underlying assumption is that optimizing network performance translates to better explanatory power for brain activity. Contrary to this assumption, our analysis of two published datasets (fMRI, EEG) reveals that the optimal alignment between brain activity and DCNNs already occurs after the first or one of the earliest iterations, and that changes in the brain-alignment are unrelated to changes in task-performance. This implies that extensive training on one task does not result in optimal brain alignment with visual cortex. It further suggests that much could be gained by aligning the training over epochs of a DCNN with learning in biological organisms.
Keywords: DCNN Encoding models Training Development