A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 750–765 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 750 A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions Takuma Udagawa1 Takato Yamazaki1 Akiko Aizawa1,2 The University of Tokyo, Tokyo, Japan1 National Institute of Informatics, Tokyo, Japan2 {takuma udagawa,takatoy,aizawa}@nii.ac.jp Abstract Recent models achieve promising results in vi- sually grounded dialogues. However, existing datasets often contain undesirable biases and lack sophisticated linguistic analyses, which make it difficult to understand how well cur- rent models recognize their precise linguis- tic structures. To address this problem, we make two design choices: first, we focus on OneCommon Corpus (Udagawa and Aizawa, 2019, 2020), a simple yet challenging com- mon grounding dataset which contains mini- mal bias by design. Second, we analyze their linguistic structures based on spatial expres- sions and provide comprehensive and reliable annotation for 600 dialogues. We show that our annotation captures important linguistic structures including predicate-argument struc- ture, modification and ellipsis. In our experi- ments, we assess the model’s understanding of these structures through reference resolution. We demonstrate that our annotation can reveal both the strengths and weaknesses of baseline models in essential levels of detail. Overall, we propose a novel framework and resource for investigating fine-grained language under- standing in visually grounded dialogues. 1 Introduction Visual dialogue is the task of holding natural, often goal-oriented conversation in a visual context (Das et al., 2017a; De Vries et al., 2017). This typically involves two types of advanced grounding: symbol grounding (Harnad, 1990), which bridges symbolic natural language and continuous visual perception, and common grounding (Clark, 1996), which refers to the process of developing mutual understandings through successive dialogues. As noted in Mon- roe et al. (2017); Udagawa and Aizawa (2019), the continuous nature of visual context introduces chal- lenging symbol grounding of nuanced and prag- matic expressions. Some further incorporate par- tial observability where the agents do not share the same context, which introduces complex misun- derstandings that need to be resolved through ad- vanced common grounding (Udagawa and Aizawa, 2019; Haber et al., 2019). Despite the recent progress on these tasks, it re- mains unclear what types of linguistic structures can (or cannot) be properly recognized by existing models for two reasons. First, existing datasets of- ten contain undesirable biases which make it possi- ble to make correct predictions without recognizing the precise linguistic structures (Goyal et al., 2017; Cirik et al., 2018; Agarwal et al., 2020). Second, existing datasets severely lack in terms of sophisti- cated linguistic analysis, which makes it difficult to understand what types of linguistic structures exist or how they affect model performance. To address this problem, we make the following design choices in this work: • We focus on OneCommon Corpus (Udagawa and Aizawa, 2019, 2020), a simple yet challeng- ing collaborative referring task under continuous and partially-observable context. In this dataset, the visual contexts are kept simple and control- lable to remove undesirable biases while enhanc- ing linguistic variety. In …

Content truncated to 500 words