The World in My Mind: Visual Dialog with Adversarial Multi-modal Feature Encoding

Proceedings of NAACL-HLT 2019, pages 2588–2598 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2588 The World in My Mind: Visual Dialog with Adversarial Multi-modal Feature Encoding Yiqun Yao123, Jiaming Xu12∗and Bo Xu1234 1Institute of Automation, Chinese Academy of Sciences (CASIA). Beijing, China 2Research Center for Brain-inspired Intelligence, CASIA 3University of Chinese Academy of Sciences 4Center for Excellence in Brain Science and Intelligence Technology, CAS. China {yaoyiqun2014,jiaming.xu,xubo}@ia.ac.cn Abstract Visual Dialog is a multi-modal task that re- quires a model to participate in a multi-turn human dialog grounded on an image, and gen- erate correct, human-like responses. In this paper, we propose a novel Adversarial Multi- modal Feature Encoding (AMFE) framework for effective and robust auxiliary training of visual dialog systems. AMFE can force the language-encoding part of a model to generate hidden states in a distribution closely related to the distribution of real-world images, resulting in language features containing general knowl- edge from both modalities by nature, which can help generate both more correct and more general responses with reasonably low time cost. Experimental results show that AMFE can steadily bring performance gains to dif- ferent models on different scales of data. Our method outperforms both the supervised learn- ing baselines and other fine-tuning methods, achieving state-of-the-art results on most met- rics of VisDial v0.5/v0.9 generative tasks. 1 Introduction In recent years, there has been a rising atten- tion in Artificial Intelligence on how to train a model to understand visual inputs from the physi- cal world, and communicate them with human lan- guage. Typical problems include Visual Question Answering (VQA) (Antol et al., 2015) and Image Captioning (Xu et al., 2015). These tasks require a model to read an image and generate a proper response, such as answering a question grounded on the image, or generating a sentence to describe the image. As a more difficult extension, Visual Dialog (De Vries et al., 2017; Das et al., 2017a; Mostafazadeh et al., 2017) is a cluster of tasks fea- turing two agents conducting a multi-turn dialog grounded on an image. A model is usually trained ∗ Corresponding Author to predict every single response of one agent in the two, based on the image and dialog history. There are also some different task settings such as directly training two agents to complete a goal- driven cooperative task such as Guessing Game (Das et al., 2017b). Tasks involving both the physical world (vi- sual images) and abstract world (languages) share a core issue: how to establish connections be- tween these two worlds, and is there a frame- work to leverage these connections for learning? Temporarily, the majority of answers are learning end-to-end models with multi-modal feature fu- sion (Kim et al., 2016; Fukui et al., 2016; Yu et al., 2018). These methods usually merge the visual and language features into rich representations containing information from both sides. Some cross-modal attention methods (Lu et al., 2016; Nam et al., 2017) formulate the visual-language connections explicitly by parameterizing the …

Content truncated to 500 words