Search Papers | Poster Sessions | All Posters
Poster B114 in Poster Session B - Thursday, August 8, 2024, 1:30 – 3:30 pm, Johnson Ice Rink
The Role of Frequency in Shaping Features from Artificial Vision Models
thomas fel1, thomas garity2, george alvarez2, thomas serre1; 1Brown University, 2Harvard University
In our study, we analyzed over 150 state-of-the-art vision models using explainability tools to see how they process high- and low-frequency features. We introduce a metric based on Attribution methods to quantify the models' dependence on high-frequency features. We found that more advanced models rely more on low-frequency features. To advance our investigation, we assessed whether more accurate models demonstrate increased reliance on phase information, which is crucial for human recognition. This was achieved by mixing the phase components of images to evaluate the models' object recognition capabilities. The findings indicate that while high-performing models are progressively depending more on phase information, they substantially lag behind human performance. Additionally, we show that models that depend on low-frequency features tend to have a shape bias, confirming a connection between frequency reliance and perception bias. Our analysis indicates that as models become more performant, their use of phase information and low-frequency features increases. However, a significant gap remains compared to human capabilities, suggesting opportunities for further enhancing model alignment through frequency analysis.
Keywords: Explainability Fourier Analysis Image Recognition Deep Learning