Search Papers | Poster Sessions | All Posters
Poster A119 in Poster Session A - Tuesday, August 6, 2024, 4:15 – 6:15 pm, Johnson Ice Rink
fMRI vision reconstruction methods robustly generalize to mental imagery
Reese Kneeland1 (), Ghislain St-Yves1, Jesse Breedlove1, Kendrick Kay1, Thomas Naselaris1; 1University of Minnesota
Fueled by recent leaps in generative AI and the release of the Natural Scenes Dataset (NSD), researchers have been able to reconstruct seen images from human brain activity with unprecedented accuracy. If it were possible to extend these visual decoding methods to mental imagery, they could potentially be useful in clinical settings, e.g., by introducing new diagnostic tools for patients with traumatic brain injuries and mental health disorders, or by providing a communication channel to patients with locked-in syndrome. We tested the application of several recent vision decoding methods to brain activity collected during a special NSD scanning session in which subjects imagined a small set of memorized target stimuli. We show that most of these methods generalize robustly to mental imagery, yielding reconstructions of mental images that human raters consistently identify as corresponding to the target stimuli in a forced-choice task. Interestingly, we find that—by this identification accuracy measure—reconstructions of imagined natural scenes are of slightly better quality than reconstructions of much simpler seen stimuli of bars and crosses. Finally, we observe a strong correlation between stimuli that reconstruct better across vision and imagery trials, suggesting that further improvements in vision decoding methods will afford improvements to mental imagery decoding.
Keywords: mental imagery decoding vision fMRI