Search Papers | Poster Sessions | All Posters
Poster C51 in Poster Session C - Friday, August 9, 2024, 11:15 am – 1:15 pm, Johnson Ice Rink
High-Fidelity Movie Reconstruction based on the fMRI decoding of Hierarchical Brain Activity
Myeonggyo Jeong1, Seok-Jun Hong1,2,3,4,5 (); 1Department of Biomedical Engineering, Sungkyunkwan University, Suwon, South Korea, 2Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, South Korea, 3Department of Intelligent Precision Healthcare Convergence, Sungkyunkwan University, Suwon, South Korea, 4Department of MetaBioHealth, Sungkyunkwan University, Suwon, South Korea, 5Center for the Developing Brain, Child Mind Institute, New York, NY, United States
Recent advances in AI-neuroscience have revolutionized our ability to decode visual images from human brain activities. Yet, reconstructing animated scenes, such as movies, remains a challenging task due to their intricate spatiotemporal dynamics. Here, we introduce a novel fMRI-based movie encoding-decoding framework, using three major self-supervised learning algorithms, that is, VideoMAE, CLIP, and Latent Diffusion Model. These algorithms, along with a simple addition to the Diffusion model, “TEmporally Smoothed LAtent Representation” (TESLAR), enabled to reconstruct the scene with photo-realistic details and enhanced temporal consistency, collectively leading to a semantically richer and natural decoding process. This result was further enhanced by our thorough investigation of brain encoding, which informed the decoding process about which brain areas have the most relevant fMRI signals to reconstruct the set of visual features. Our framework has a high potential to reveal key representational mechanisms underlying complex perceptual processes in the human brain.
Keywords: Brain Encoding Brain Decoding Generative AI Cortical hierarchy