Search Papers | Poster Sessions | All Posters
Poster B76 in Poster Session B - Thursday, August 8, 2024, 1:30 – 3:30 pm, Johnson Ice Rink
Measuring Alignment between Human and Artificial Intelligence with Representational Similarity Analysis
Mattson Ogg1 (), Michael Wolmetz1; 1Johns Hopkins University Applied Physics Laboratory
Large Language Models (LLMs) are improving at an incredible rate. With increasing scale comes emergent properties, including an ostensibly human-like understanding of the world. However, it is difficult to assess how these models process and represent information and it is not clear how best to measure their similarities with humans. To help meet this need, we developed a generalizable behavioral task for LLMs (sometimes called a “Turing Experiment”) based around pairwise behavioral ratings to facilitate a representational similarity analysis (RSA) that measures alignment among LLM and human agents. Using this method, which we refer to as “Turing RSA,” we quantified how aligned the similarity ratings that different LLMs provided for a well-studied set of stimuli from the cognitive neuroscience literature were to human responses at a group and individual level. We found GPT-4 to be the best current proxy of human behavior among its family of models across text and image modalities, but that the inter-individual variability among human participants is hard to reproduce with LLMs. We show that RSA helps us understand how LLMs encode knowledge about the world, examine the variability among agents, and measure their representational alignment with humans.
Keywords: Large Language Models Explainability Representational Similarity Artificial Intelligence