A Visually-grounded First-person Dialogue Dataset with Verbal and Non-verbal Responses

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3299–3310, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3299 A Visually-grounded First-person Dialogue Dataset with Verbal and Non-verbal Responses Hisashi Kamezawa1 Noriki Nishida2 Nobuyuki Shimizu3 Takashi Miyazaki3 Hideki Nakayama1 1 The University of Tokyo 2 RIKEN Center for Advanced Intelligence Project (AIP) 3 Yahoo Japan Corporation hisas@nlab.ci.i.u-tokyo.ac.jp noriki.nishida@riken.jp {nobushim, takmiyaz}@yahoo-corp.jp nakayama@ci.i.u-tokyo.ac.jp Abstract In real-world dialogue, first-person visual in- formation about where the other speakers are and what they are paying attention to is crucial to understand their intentions. Non-verbal re- sponses also play an important role in social in- teractions. In this paper, we propose a visually- grounded first-person dialogue (VFD) dataset with verbal and non-verbal responses. The VFD dataset provides manually annotated (1) first-person images of agents, (2) utterances of human speakers, (3) eye-gaze locations of the speakers, and (4) the agents’ verbal and non- verbal responses. We present experimental re- sults obtained using the proposed VFD dataset and recent neural network models (e.g., BERT, ResNet). The results demonstrate that first- person vision helps neural network models cor- rectly understand human intentions, and the production of non-verbal responses is a chal- lenging task like that of verbal responses. Our dataset is publicly available1. 1 Introduction In recent years, visually-grounded dialogue sys- tems have attracted increasing attention (Zhu et al., 2016; Ben-Youssef et al., 2017; Liao et al., 2018; Kottur et al., 2018). For example, Huber et al. (2018) developed an image-grounded conversa- tional agent that uses visual sentiment, facial ex- pression, and scene features, and Mostafazadeh et al. (2017) constructed the publicly available IGC dataset, which comprises image-grounded conver- sations. Although these studies and resources have been shown to be useful, there are currently two limi- tations. First, in image-grounded dialogue tasks, 1https://randd.yahoo.co.jp/en/softwaredata U: これのＬはないのかしら V: 同じ服がたくさんあるからどれかはLじゃないかな N: 同じ服のサイズをチェックする ————————————————————————– U: I wonder if there is an L for this. V: We have a lot of the same clothes, so I’m guessing one of them is an L. N: Check out the same clothing size. Figure 1: Example of proposed VFD dataset. “U”, “V”, and “N” denote a human utterance, the agent’s verbal response, and the agent’s non-verbal response (i.e., ac- tion), respectively. All utterances and responses are represented in Japanese. English translations are added below for easier understanding. The red line links the eyes to the gaze location. human speakers do not appear in the agents’ vision because images are used as the topic of conversa- tion, and the speakers are required to discuss the input image. However, in real-world dialogue sce- narios, first-person visual information about where the human speaker is and what they are paying at- tention to is crucial for agents to understand human intentions. To understand this, we show an exam- ple in Figure 1. Without the first-person image, it is difficult for the agent to recognize that the pro- noun “this” in the human utterance (U) refers to the article of yellow clothing rather than any other products (e.g., …

Content truncated to 500 words