Maria: A Visual Experience Powered Conversational Agent

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5596–5611 August 1–6, 2021. ©2021 Association for Computational Linguistics 5596 Maria: A Visual Experience Powered Conversational Agent Zujie Liang1∗ † Huang Hu2† Can Xu2 Chongyang Tao2 Xiubo Geng2 Yining Chen2 Fan Liang1 Daxin Jiang2‡ 1School of Electronics and Information Technology, Sun Yat-sen University, Guangzhou, China 2Microsoft STCA NLP Group, Beijing, China 1{liangzj9@mail2.sysu.edu.cn, isslf@mail.sysu.edu.cn} 2{huahu,caxu,chotao,xigeng,yinichen,djiang}@microsoft.com Abstract Arguably, the visual perception of conversa- tional agents to the physical world is a key way for them to exhibit the human-like intelligence. Image-grounded conversation is thus proposed to address this challenge. Existing works fo- cus on exploring the multimodal dialog mod- els that ground the conversation on a given image. In this paper, we take a step further to study image-grounded conversation under a fully open-ended setting where no paired dia- log and image are assumed available. Specifi- cally, we present Maria, a neural conversation agent powered by the visual world experiences which are retrieved from a large-scale image index. Maria consists of three flexible compo- nents, i.e., text-to-image retriever, visual con- cept detector and visual-knowledge-grounded response generator. The retriever aims to re- trieve a correlated image to the dialog from an image index, while the visual concept detec- tor extracts rich visual knowledge from the im- age. Then, the response generator is grounded on the extracted visual knowledge and dialog context to generate the target response. Ex- tensive experiments demonstrate Maria outper- forms previous state-of-the-art methods on au- tomatic metrics and human evaluation, and can generate informative responses that have some visual commonsense of the physical world. 1 Introduction Building intelligent conversational agents that can not only converse freely with human but also have the ability to perceive the physical world, has been one of the longest standing goals of natural lan- guage processing (NLP) and artificial intelligence (AI). Although the recent large-scale conversation models trained on text-only corpora, such as Meena ∗Work performed during the internship at Microsoft. †Equal contribution. ‡Corresponding author. Human-A: Hey! How was your vacation? Human-B: Awesome! I had a good time with my friends in Hawaii, the beaches are very beautiful there. Human-A: Cool! did you play beach volleyball with your friends? (Human-A: Cool, have you had a BBQ with your friends on the beach? The grilled fish was great!) Human-B: Nope, but it sounds great. Maybe next time. Figure 1: An example of human conversations. When human-B talks about vacation on the beach of Hawaii, human-A recalls his/her past experience of playing vol- leyball or having BBQ on the beach. (Adiwardana et al., 2020), Blender (Roller et al., 2020) and DialoGPT (Zhang et al., 2020), have shown the compelling performance, they are still lack of the perception ability to our physical world. A recent study (Bisk et al., 2020) points out the suc- cessful linguistic communication relies on a shared experience of the world that makes language re- ally meaningful. The visual perception is …

Content truncated to 500 words