Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2236 - 2250 December 7-11, 2022 ©2022 Association for Computational Linguistics Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality Anuj Diwan∗,♢, Layne Berry∗,♢, Eunsol Choi♢, David Harwath♢, Kyle Mahowald♡ ♢Department of Computer Science ♡Department of Linguistics The University of Texas at Austin {anuj.diwan, layne.berry, eunsol, harwath, mahowald}@utexas.edu Abstract Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset (Thrush et al., 2022), which challenges models to match paired im- ages and English captions, with items con- structed to overlap lexically but differ in mean- ing (e.g., “there is a mug in some grass” vs. “there is some grass in a mug”). By annotat- ing the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus ob- jects in low-resolution images. In this paper, we identify the dataset’s main challenges through a suite of experiments on related tasks (prob- ing task, image retrieval task), data augmenta- tion, and manual inspection of the dataset. Our analysis suggests that a main challenge in vi- suolinguistic models may lie in fusing visual and textual representations, rather than in com- positional language understanding. We release our annotation and code at https://github. com/ajd12342/why-winoground-hard. 1 Introduction Despite the success of large pretrained transformer models on a wide variety of tasks, the extent to which they are compositional (e.g., Kim and Linzen, 2020; Soulos et al., 2020; Hewitt and Man- ning, 2019; Sinha et al., 2021a; Clouatre et al., 2021) and grounded (Bender and Koller, 2020; Bisk et al., 2020) is debated. Taking compositional- ity and groundedness as key desiderata, the recent Winoground dataset (Thrush et al., 2022) provides a clever way to test multimodal vision and language models. Given two images and two captions, the goal is to pair them correctly. The key insight is *Co-first authors contributed equally. Order determined by coin flip. the cat on the left of the photo has its right paw ahead of its left the cat on the left of the photo has its left paw ahead of its right original NonCompositional AmbiguouslyCorrect VisuallyDifficult ✓ UnusualImage UnusualText ComplexReasoning ✓ (A) The original Winoground task… the cat on the left hand side of the photo has the right paw in front of the left the cat on the left of the photo has ahead of its right its left paw variants the animal on the left of the photo has its right paw ahead of its left the cat that is on the left of the photo has its left paw ahead of its right (B) With new tags… (Section 4) (C) Evaluated with caption variants! (Section 5) Figure 1: Extending the (A) original Winoground task, which presents a minimal semantic pair of image cap- tions and their corresponding …

Content truncated to 500 words