Search Papers | Poster Sessions | All Posters
Poster C49 in Poster Session C - Friday, August 9, 2024, 11:15 am – 1:15 pm, Johnson Ice Rink
In search of the embgram: forming episodic representations in a deep learning model
Chad DeChant1 (), Iretiayo Akinola2, Daniel Bauer1; 1Columbia University, 2NVIDIA
Enabling episodic memory in a robot or other AI agent would lead to better functioning AI as well as creating opportunities for modeling theories of human memory. An important step toward such an episodic memory system is developing representations that can be used to accurately recover information about past events. We introduce a method to obtain such representations with an artificial neural network model. To study episodes of realistic length, we utilize ego-centric video data from a dataset of an agent performing household chores in a simulated environment. In the first training phase, we use a transformer model to encode frames from this video into compact vector embeddings — embgrams. Next, using the embgrams as input, a second transformer model is fine-tuned to provide natural language descriptions and answers to questions about the episodes. Our results show that the embgrams facilitate retrieval of episode-related information. Importantly, we find that the usefulness of the embgrams as stores of information significantly depends on the task the encoding transformer performs during their generation.
Keywords: episodic memory attention representation learning