Multi-Modal Open-Domain Dialogue

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4863–4883 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4863 Multi-Modal Open-Domain Dialogue Kurt Shuster∗, Eric Michael Smith∗, Da Ju, Jason Weston Facebook AI Research {kshuster,ems,daju,jase}@fb.com Abstract Recent work in open-domain conversational agents has demonstrated that significant im- provements in humanness and user preference can be achieved via massive scaling in both pre-training data and model size (Adiwardana et al., 2020; Roller et al., 2020). However, if we want to build agents with human-like abil- ities, we must expand beyond handling just text. A particularly important topic is the ability to see images and communicate about what is perceived. With the goal of getting humans to engage in multi-modal dialogue, we investigate combining components from state-of-the-art open-domain dialogue agents with those from state-of-the-art vision models. We study incorporating different image fusion schemes and domain-adaptive pre-training and fine-tuning strategies, and show that our best resulting model outperforms strong existing models in multi-modal dialogue while simul- taneously performing as well as its predeces- sor (text-only) BlenderBot (Roller et al., 2020) in text-based conversation. We additionally in- vestigate and incorporate safety components in our final model, and show that such efforts do not diminish model performance with respect to human preference. 1 Introduction An important goal of artificial intelligence is the construction of open-domain conversational agents that can engage humans in discourse. Indeed, the future of human interaction with AI is predicated on models that can exhibit a number of different conversational skills over the course of rich dia- logue. Much recent work has explored building and training dialogue agents that can blend such skills throughout natural conversation, with the ultimate goal of providing an interesting and engrossing ex- perience for humans (Smith et al., 2020; Shuster et al., 2019b). Coupled with the advancement of ∗Joint First Authors. Figure 1: Cherry picked conversation between a pa- per author (right) and our MMB DegenPos model (left). More sample conversations are in the appendix. large-scale model training schemes, such models are becoming increasingly human-like and engag- ing (Zhang et al., 2020; Adiwardana et al., 2020; Roller et al., 2020). In order to better approach human-like ability, however, it is necessary that agents can converse with both textual and visual context, similarly to how humans interact in the real world; indeed, com- munication grounded in images is naturally engag- ing to humans (Hu et al., 2014). Recent efforts have gone beyond classical, fact-based tasks such as image captioning or visual question answering (Antol et al., 2015; Das et al., 2017a) to produce models that can respond and communicate about images in the flow of natural conversation (Shuster et al., 2020, 2019b). In this work, we explore the extension of large- scale conversational agents to image-based dia- 4864 logue. We combine representations from image- based models that have been trained on object de- tection tasks (Lu et al., 2020, 2019) with represen- tations from Transformers with billions of param- eters pre-trained on massive (text-only) …

Content truncated to 500 words