COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9824–9846 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9824 COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images Ben Bogin1 Shivanshu Gupta2 Matt Gardner3 Jonathan Berant1,3 1Tel-Aviv University 2University of California Irvine 3Allen Institute for AI {ben.bogin,joberant}@cs.tau.ac.il, shivag5@uci.edu, mattg@allenai.org Abstract While interest in models that generalize at test time to new compositions has risen in recent years, benchmarks in the visually-grounded domain have thus far been restricted to syn- thetic images. In this work, we propose COVR, a new test-bed for visually-grounded compositional generalization with real images. To create COVR, we use real images anno- tated with scene graphs, and propose an al- most fully automatic procedure for generating question-answer pairs along with a set of con- text images. COVR focuses on questions that require complex reasoning, including higher- order operations such as quantification and ag- gregation. Due to the automatic generation process, COVR facilitates the creation of com- positional splits, where models at test time need to generalize to new concepts and compo- sitions in a zero- or few-shot setting. We con- struct compositional splits using COVR and demonstrate a myriad of cases where state-of- the-art pre-trained language-and-vision mod- els struggle to compositionally generalize. 1 Introduction Models for natural language understanding (NLU) have exhibited remarkable generalization abilities on many tasks, when the training and test data are sampled from the same distribution. But such mod- els still lag far behind humans when asked to gener- alize to an unseen combination of known concepts, and struggle to learn concepts for which only few examples are provided (Finegan-Dollak et al., 2018; Bahdanau et al., 2019a). Humans, conversely, do this effortlessly: for example, once humans learn the meaning of the quantifier “all”, they can eas- ily understand the utterance “all cheetahs have spots” if they know what “cheetahs” and “spots” mean (Chomsky, 1957; Montague, 1970; Fodor and Pylyshyn, 1988). This ability, termed composi- tional generalization, is crucial for building models that generalize to new settings (Lake et al., 2018). Q: All skateboards are blue A: TRUE Q: There is a man reading a book and wearing eyeglasses A: FALSE Q: All women are holding a purse and are sitting on a bench A: FALSE Train Test ★ Figure 1: Compositional generalization in the VQA setup. COVR enables the creation of compositional splits such as the one depicted here, where quantifica- tion appears with conjunction only in the test set. In recent years, multiple benchmarks have been created, illustrating that current NLU models fail to generalize to new compositions. However, these benchmarks focused on semantic parsing, the task of mapping natural language utterances to logical forms (Lake and Baroni, 2018; Kim and Linzen, 2020; Keysers et al., 2020). Visual question an- swering (VQA) is arguably a harder task from the perspective of compositional generalization, since the model needs to learn to compositionally “ex- ecute” the meaning of the question over images, without being exposed to an explicit …

Content truncated to 500 words