Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2751–2767 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2751 Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision Sandro Pezzelle1, Claudio Greco2, Greta Gandolfi2, Eleonora Gualdoni2, Raffaella Bernardi2,3 1Institute for Logic, Language and Computation, University of Amsterdam 2CIMeC, 3DISI, University of Trento s.pezzelle@uva.nl, {greta.gandolfi|eleonora.gualdoni}@studenti.unitn.it, {claudio.greco|raffaella.bernardi}@unitn.it Abstract This paper introduces BD2BB, a novel lan- guage and vision benchmark that requires mul- timodal models combine complementary infor- mation from the two modalities. Recently, im- pressive progress has been made to develop universal multimodal encoders suitable for vir- tually any language and vision tasks. However, current approaches often require them to com- bine redundant information provided by lan- guage and vision. Inspired by real-life com- municative contexts, we propose a novel task where either modality is necessary but not suf- ficient to make a correct prediction. To do so, we first build a dataset of images and cor- responding sentences provided by human par- ticipants. Second, we evaluate state-of-the-art models and compare their performance against human speakers. We show that, while the task is relatively easy for humans, best-performing models struggle to achieve similar results. 1 Introduction Human communication, in real-life situations, is multimodal (Kress, 2010): To convey and under- stand a message uttered in natural language, people build on what is present in the multimodal con- text surrounding them. As such, speakers do not need to “repeat” something that is already provided by the environment; similarly, listeners leverage information from various modalities, such as vi- sion, to interpret the linguistic message. Integrat- ing information from multiple modalities is indeed crucial for attention and perception (Partan and Marler, 1999) since combined information from concurrent modalities can give rise to different mes- sages (McGurk and MacDonald, 1976). The argument that language and vision convey different, possibly complementary aspects of mean- ing has been largely made to motivate the need for multimodal semantic representations of words (Ba- roni, 2016; Beinborn et al., 2018). However, com- putational approaches to language and vision typi- cally do not fully explore this complementarity. To illustrate, given an image (e.g., the one depicted in Figure 1), popular tasks involve describing it in natural language, e.g., “A tennis player about to hit the ball” (Image Captioning; see Bernardi et al., 2016); answering questions that are grounded in it, e.g., Q: “What sport is he playing?”, A: “Ten- nis” (Visual Question Answering; see Antol et al., 2015); having a dialogue on its entities, e.g., Q: “Is the person holding a racket?”, A: “Yes.” (visually- grounded dialogue; see De Vries et al., 2017; Das et al., 2017). While all these tasks challenge mod- els to perform visual grounding, i.e., an effective alignment of language and vision, none of them require a genuine combination of complementary information provided by the two modalities. All the information is fully available in the visual scene, and language is used to describe or retrieve it. In this work, we …

Content truncated to 500 words