Multimodal Dialogue Response Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2854 - 2866 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multimodal Dialogue Response Generation Qingfeng Sun1 Yujing Wang2 Can Xu1 Kai Zheng1 Yaming Yang2 Huang Hu1 Fei Xu1 Jessica Zhang1 Xiubo Geng1 Daxin Jiang1∗ 1Microsoft STC Asia 2Microsoft Research Asia {qins,yujwang,caxu,zhengkai,yayaming,huahu,fexu, jessicaz,xigeng,djiang}@microsoft.com Abstract Responsing with image has been recognized as an important capability for an intelligent con- versational agent. Yet existing works only fo- cus on exploring the multimodal dialogue mod- els which depend on retrieval-based methods, but neglecting generation methods. To fill in the gaps, we first present a new task: multi- modal dialogue response generation (MDRG) - given the dialogue context, one model needs to generate a text or an image as response. Learn- ing such a MDRG model often requires multi- modal dialogues containing both texts and im- ages which are difficult to obtain. Motivated by the challenge in practice, we consider MDRG under a natural assumption that only limited training examples are available. Under such a low-resource setting, we devise a novel conver- sational agent, Divter, in order to isolate param- eters that depend on multimodal dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of text-only dialogues and text- image pairs respectively, then the whole param- eters can be well fitted using just a few training examples. Extensive experiments demonstrate our method achieves state-of-the-art results in both automatic and human evaluation, and can generate informative text and high-resolution image responses. 1 Introduction With the development of instant messaging tech- nology in the recent decades, the intermediary of online conversation has also changed from pure text to a variety of visual modalities (e.g., image, gif animation, short video). Similar to communicating by the messenger tools (e.g., Facebook, WhatsApp, WeChat) in reality, an excellent intelligent conver- sational agent should not only be able to converse freely with plain text, but also have the ability to perceive and share the real visual physical world. ∗ Corresponding author. Hey Amy, how's your day going? I'm on vacation in Venezia, falling in love with the colors of Burano. And the best dinner at Gatto Nero! Wow, it looks so beautiful! Envy you. So what’s about you today Tom? Haha, just went outdoors with my family. What a warm scene. Your girls are pretty happy. Figure 1: An example of human conversations. They are talking about vacation and outdoors with both text and various images. Although recently some large-scale pre-trained text-only dialogue generation models, such as Di- aloGPT (Zhang et al., 2020), Blender (Roller et al., 2021), Meena (Adiwardana et al., 2020), have shown excellent performance, they still cannot rely exclusively on plain text to completely simulate the rich experience of visual perception. Recently, vari- ous vision-language tasks have been introduced and attracted widespread attention, such as visual ques- tion answering (Ren et al., 2015; Lu et al., 2016; Anderson …

Content truncated to 500 words