Keynote Lecture: Leyla Isik

Seeing Social Interactions

Leyla Isik, Johns Hopkins University
Clare Boothe Luce Assistant Professor

Humans see the world in rich social detail. We effortlessly recognize not only objects and people in our environment, but also social interactions between people. The ability to perceive and understand others’ interactions is critical to function in our social world, yet the underlying neural computations remain poorly understood. In this talk, I will first argue that social interaction perception should be studied with the same computational vision tools that are now widely applied to other areas of vision, like scene and object recognition. I will then present new research using a large-scale, naturalistic video dataset and condition-rich fMRI experiment, demonstrating that social interaction information is extracted hierarchically by the visual system along the recently proposed lateral visual pathway. In ongoing work with this same dataset, we find that unlike static scene and object vision, current AI vision models do a poor job of matching human behavior and neural responses to dynamic, social scenes. To help close this gap, we developed a novel graph neural network model, SocialGNN, that instantiates insights from cognitive (neuro)science. SocialGNN reproduces human judgments of social interactions in both controlled and natural videos using only visual information, without any explicit model of agents’ minds or the physical world. Critically, the model’s relational, graph-based structure and processing are required for accurate social interaction recognition. Preliminary data also show that SocialGNN matches neural responses along the lateral visual pathway. Together, this research suggests that social interaction recognition is a core human ability that relies on specialized, structured visual representations.