DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5651–5665 August 1–6, 2021. ©2021 Association for Computational Linguistics 5651 DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue Hung Le‡§∗, Chinnadhurai Sankar†, Seungwhan Moon†, Ahmad Beirami†, Alborz Geramifard†, Satwik Kottur† †Facebook {chinnadhurai, shanemoon, beirami, alborzg, skottur}@fb.com ‡Singapore Management University §Institute for Infocomm Research, A*STAR hungle.2018@smu.edu.sg Abstract A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations. Building such dialogue systems is a challenging problem, in- volving various reasoning types on both vi- sual and language inputs. Existing bench- marks do not have enough annotations to thor- oughly analyze dialogue systems and under- stand their capabilities and limitations in iso- lation. These benchmarks are also not ex- plicitly designed to minimise biases that mod- els can exploit without actual reasoning. To address these limitations, in this paper, we present DVD, a Diagnostic Dataset for Video- grounded Dialogues. The dataset is designed to contain minimal biases and has detailed an- notations for the different types of reasoning over the spatio-temporal space of video. Dia- logues are synthesized over multiple question turns, each of which is injected with a set of cross-turn semantic relationships. We use DVD to analyze existing approaches, provid- ing interesting insights into their abilities and limitations. In total, DVD is built from 11k CATER synthetic videos and contains 10 in- stances of 10-round dialogues for each video, resulting in more than 100k dialogues and 1M question-answer pairs. Our code and dataset are publicly available1. 1 Introduction Research in visual question answering (VQA) aims to develop intelligent systems that can reason and answer questions about visual information. Ear- lier datasets have been introduced to study this problem, focusing on images as the visual input (Antol et al., 2015; Gao et al., 2015; Malinowski ∗Work done when HL was a research intern at Facebook. 1github.com/facebookresearch/ DVDialogues Q1: until the end of the cube 's rotation , what types of actions does the big thing undertake the most ? A1: flying Q2: during the same time period , how many sliding objects are there ? A2: 2 Q3: among them , there is a ball . during the whole video , what type of action does it undertake second ? A3: no action Q4: how about up until now ? A4: sliding Q5: during the red thing 's last slide , how many things are behind the earlier mentioned large object ? A5: 2 Q6: how about left of it ? A6: 0 ... t1 t2 t3 t4 t0 T Object A: t1 t3 Object B: t2 t4 t5 t6 t5 t6 slides flies rotates Figure 1: Example DVD dialogue: We demonstrate an example dialogue in DVD that tests various aspects, in- cluding action recognition, temporal reasoning, spatial reasoning, video interval tracking, and dialogue object tracking. …

Content truncated to 500 words