A Corpus for Reasoning about Natural Language Grounded in Photographs

A Corpus for Reasoning About Natural Language Grounded in Photographs Alane Suhr‡∗, Stephanie Zhou†,∗ Ally Zhang‡, Iris Zhang‡, Huajun Bai‡, and Yoav Artzi‡ ‡Cornell University Department of Computer Science and Cornell Tech New York, NY 10044 {suhr, yoav}@cs.cornell.edu {az346, wz337, hb364}@cornell.edu †University of Maryland Department of Computer Science College Park, MD 20742 stezhou@cs.umd.edu Abstract We introduce a new dataset for joint reason- ing about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data con- tains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language cap- tion is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Quali- tative analysis shows the data requires compo- sitional joint reasoning, including about quan- tities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning meth- ods shows the data presents a strong challenge. 1 Introduction Visual reasoning with natural language is a promising avenue to study compositional seman- tics by grounding words, phrases, and complete sentences to objects, their properties, and rela- tions in images. This type of linguistic reason- ing is critical for interactions grounded in visually complex environments, such as in robotic appli- cations. However, commonly used resources for language and vision (e.g., Antol et al., 2015; Chen et al., 2016) focus mostly on identification of ob- ject properties and few spatial relations (Section 4; Ferraro et al., 2015; Alikhani and Stone, 2019). This relatively simple reasoning, together with bi- ases in the data, removes much of the need to consider language compositionality (Goyal et al., 2017). This motivated the design of datasets that require compositional1 visual reasoning, including ∗Contributed equally. † Work done as an undergraduate at Cornell University. 1In parts of this paper, we use the term compositional dif- ferently than it is commonly used in linguistics to refer to reasoning that requires composition. This type of reasoning often manifests itself in highly compositional language. The left image contains twice the number of dogs as the right image, and at least two dogs in total are standing. One image shows exactly two brown acorns in back-to-back caps on green foliage. Figure 1: Two examples from NLVR2. Each caption is paired with two images.2 The task is to predict if the caption is True or False. The examples require addressing challenging semantic phenomena, includ- ing resolving twice ...as to counting and comparison of objects, and composing cardinality constraints, such as at least two dogs in total and exactly two.3 NLVR (Suhr et al., 2017) and CLEVR (Johnson et al., 2017a,b). These datasets use synthetic im- ages, synthetic language, or both. The result is a limited representation of linguistic challenges: synthetic languages are inherently of bounded ex- pressivity, and synthetic visual input entails lim- ited lexical and semantic diversity. We address these limitations with Natural Lan- guage Visual Reasoning for Real (NLVR2), a new dataset for reasoning about natural language …

Content truncated to 500 words