CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog

Proceedings of NAACL-HLT 2019, pages 582–595 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 582 CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog Satwik Kottur1, Jos´e M.F. Moura1, Devi Parikh2,3, Dhruv Batra2,3, Marcus Rohrbach2 1Carnegie Mellon University, 2Facebook AI Research, 3Georgia Institute of Technology {skottur,moura}@andrew.cmu.edu,{parikh,dbatra}@gatech.edu,mrf@fb.com Abstract Visual Dialog is a multimodal task of an- swering a sequence of questions grounded in an image (using the conversation history as context). It entails challenges in vision, language, reasoning, and grounding. How- ever, studying these subtasks in isolation on large, real datasets is infeasible as it requires prohibitively-expensive complete annotation of the ‘state’ of all images and dialogs. We develop CLEVR-Dialog, a large diagnos- tic dataset for studying multi-round reasoning in visual dialog. Specifically, we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset. This combination results in a dataset where all aspects of the visual dialog are fully annotated. In total, CLEVR-Dialog contains 5 instances of 10-round dialogs for about 85k CLEVR im- ages, totaling to 4.25M question-answer pairs. We use CLEVR-Dialog to benchmark perfor- mance of standard visual dialog models; in particular, on visual coreference resolution (as a function of the coreference distance). This is the first analysis of its kind for visual dia- log models that was not possible without this dataset. We hope the findings from CLEVR- Dialog will help inform the development of fu- ture models for visual dialog. Our code and dataset are publicly available1. 1 Introduction The focus of this work is on intelligent systems that can see (perceive their surroundings through vision), talk (hold a visually grounded dialog), and reason (store entities in memory as a dialog pro- gresses, refer back to them as appropriate, count, compare, etc.). Recent works have begun studying such systems under the umbrella of Visual Dialog (Das et al., 2017a; de Vries et al., 2017), where 1https://github.com/satwikkottur/clevr-dialog an agent must answer a sequence of questions grounded in an image. As seen in Fig. 1, this entails challenges in – vision (e.g., identifying objects and their attributes in the image), language/reasoning (e.g., keeping track of and referencing previous conversation via memory), and grounding (e.g., grounding textual entities in the image). In order to train and evaluate agents for Visual Dialog, Das et al. (2017a) collected a large dataset of human-human dialog on real images collected between pairs of workers on Amazon Mechani- cal Turk (AMT). While such large-scale realistic datasets enable new lines of research, it is difficult to study the different challenges (vision, language, reasoning, grounding) in isolation or to break down the performance of systems over different chal- lenges to identify bottlenecks, because that would require prohibitively-expensive complete annota- tion of the ‘state’ of all images and dialogs (all entities, coreferences, etc.). In this work, we draw inspiration from John- son et al. (2017), and develop a large diagnos- tic dataset—CLEVR-Dialog—for studying and benchmarking multi-round reasoning in visually- grounded dialog. Each CLEVR image …

Content truncated to 500 words