Inducing Semantic Roles Without Syntax

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 436–446 August 1–6, 2021. ©2021 Association for Computational Linguistics 436 Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation Feilong Chen, Fandong Meng, Xiuyi Chen, Peng Li, Jie Zhou Pattern Recognition Center, WeChat AI, Tencent Inc, Beijing, China {ivess.chan,hugheren.chan}@gmail.com {fandongmeng,patrickpli,withtomzhou}@tencent.com Abstract Visual dialogue is a challenging task since it needs to answer a series of coherent ques- tions on the basis of understanding the vi- sual environment. Previous studies focus on the implicit exploration of multimodal co- reference by implicitly attending to spatial image features or object-level image features but neglect the importance of locating the ob- jects explicitly in the visual content, which is associated with entities in the textual con- tent. Therefore, in this paper we propose a Multimodal Incremental Transformer with Visual Grounding, named MITVG, which con- sists of two key parts: visual grounding and multimodal incremental transformer. Visual grounding aims to explicitly locate related ob- jects in the image guided by textual entities, which helps the model exclude the visual con- tent that does not need attention. On the ba- sis of visual grounding, the multimodal incre- mental transformer encodes the multi-turn dia- logue history combined with visual scene step by step according to the order of the dialogue and then generates a contextually and visually coherent response. Experimental results on the VisDial v0.9 and v1.0 datasets demonstrate the superiority of the proposed model, which achieves comparable performance. 1 Introduction Recently, there is increasing interest in vision- language tasks, such as image caption (Xu et al., 2015; Anderson et al., 2016, 2018; Cornia et al., 2020) and visual question answering (Ren et al., 2015a; Gao et al., 2015; Lu et al., 2016; Ander- son et al., 2018). In the real world, our conver- sations (Chen et al., 2020b, 2019) usually have multiple turns. As an extension of conventional single-turn visual question answering, Das et al. (2017) introduce a multi-turn visual question an- swering task named visual dialogue, which aims to Q1: how many people ? Q2: is anyone holding a frisbee ? Q3: is the coach on the right ? Q4: are they wearing matching uniforms ? Caption: there is a frisbee team with their coach taking a team photo A1: 7 people A2: yes A3: yes, on the far right A4: all except the coach Figure 1: An example of visual dialogue. The color in text background corresponds to the same color box in the image, which indicates the same entity. Our model firstly associates textual entities with objects explicitly and then gives contextually and visually coherent an- swers to contextual questions. explore the ability of an AI agent to hold a mean- ingful multi-turn dialogue with humans in natural language about visual content. Visual dialogue (Agarwal et al., 2020; Wang et al., 2020; Qi et al., 2020; Murahari et al., 2020) requires agents to give a response on the basis of understanding both visual and textual content. One of the key challenges in …

Content truncated to 500 words