Search Papers | Poster Sessions | All Posters
Poster C53 in Poster Session C - Friday, August 9, 2024, 11:15 am – 1:15 pm, Johnson Ice Rink
What is a good model for brain encoding in a videogame task ?
François Paugam1,2,3 (), Guillaume Lajoie1,2, Pierre Bellec1,3; 1University of Montreal, 2Mila - Quebec AI Institute, 3Centre de Recherche de l’Institut Universitaire de Gériatrie de Montréal
Videogames represent a promising experimental paradigm for neuroscientists to study active tasks in complex environments. However, interpreting brain dynamics in such complex environments is challenging, though a recent approach is to use brain encoding, i.e. quantify the similarities in activity between the brain and an artificial neural network. A wide range of modelling approaches could potentially be used to encode brain activity in videogames. In this work we compare three machine learning models trained with different objective functions to encode fMRI data collected on 5 subjects playing Super Mario Bros: (1) PPO was trained with reinforcement learning to play the game from video frames; (2) VideoGPT was trained through predictive coding on videos of human gameplay; (3) ResNet was trained for image classification in a diverse set of natural images. All three models produced qualitatively similar brain encoding maps on the levels used for training, though overall ResNet had better brain encoding accuracy and generalised better to new levels. As VideoGPT and PPO were trained from scratch on videogame data, they demonstrate the feasibility of future experiments to explain brain activity during videogames while carefully controlling the nature and size of data used for training.
Keywords: brain encoding artificial neural networks fMRI videogames