Does language help generalization in vision models?

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 171–182 November 10–11, 2021. ©2021 Association for Computational Linguistics 171 Does language help generalization in vision models? Benjamin Devillers1, Bhavin Choksi2, Romain Bielawski1, Rufin VanRullen1,2 1 Artificial and Natural Intelligence Toulouse Institute, Université de Toulouse, France {firstname.lastname}@univ-tlse3.fr 2 CerCO, CNRS UMR5549, Toulouse {firstname.lastname}@cnrs.fr Abstract Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zero- shot and transfer learning settings. This could imply that linguistic or “semantic grounding” confers additional generalization abilities to the visual feature space. Here, we system- atically evaluate various multimodal architec- tures and vision-only models in terms of unsu- pervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no ad- ditional generalization capability compared to standard supervised visual training. We con- clude that work is still required for semantic grounding to help improve vision models. 1 Introduction Learning vision models using language supervi- sion has gained popularity (Quattoni et al., 2007; Srivastava and Salakhutdinov, 2012; Frome et al., 2013; Joulin et al., 2016; Pham et al., 2019; De- sai and Johnson, 2021; Hu and Singh, 2021; Rad- ford et al., 2021; Sariyildiz et al., 2020) for two main reasons: firstly, vision-language training al- lows to build massive training datasets from readily available online data, without manual annotation; secondly, language provides additional semantic information that cannot be inferred from vision- only datasets, and this could help with semantic grounding of visual features. Recently (Radford et al., 2021) introduced CLIP, a language and vision model that shows outstand- ing zero-shot learning capabilities on many tasks, and compelling transfer-learning abilities. A re- cent report (Goh et al., 2021) showed that CLIP produces neural selectivity patterns comparable to “multimodal” concept cells observed in the human brain (Quiroga et al., 2005; Reddy and Thorpe, 2014). From these results, it is tempting to assume Figure 1: Size of the training dataset used by the mod- els (number of images, in log scale). ICMLM and VirTex are trained on CoCo, TSM on HowTo100M, CLIP on a (not publicly available) scrape of the inter- net, RN50 is trained on ImageNet-1k, the AR models and SIN models are trained on augmented versions of ImageNet-1k and BiT-M is trained on ImageNet-21k. that CLIP’s generalization properties stem from semantic grounding provided by the joint vision- language training. Here, we show that CLIP and other vision- language models do not perform better than vision- only, fully supervised models on a number of gener- alization settings and datasets. Representation sim- ilarity (Kriegeskorte et al., 2008) analysis reveals that the multimodal representations that emerge through vision-language training are different from both linguistic and visual representations–and thus possibly unsuitable for transfer-learning to new vi- sual tasks. In conclusion, additional work on lin- guistic grounding is still needed, if it is to improve generalization capabilities of vision models. We provide our code for reproducibility1. 2 Models We use …

Content truncated to 500 words