Internet-Augmented Dialogue Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8460 - 8478 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Internet-Augmented Dialogue Generation Mojtaba Komeili Facebook AI Research Labs komeili@fb.com Kurt Shuster Facebook AI Research Labs kshuster@fb.com Jason Weston Facebook AI Research Labs jase@fb.com Abstract The largest store of continually updating knowl- edge on our planet can be accessed via in- ternet search. In this work we study giving access to this information to conversational agents. Large language models, even though they store an impressive amount of knowledge within their weights, are known to hallucinate facts when generating dialogue (Shuster et al., 2021); moreover, those facts are frozen in time at the point of model training. In contrast, we propose an approach that learns to generate an internet search query based on the context, and then conditions on the search results to fi- nally generate a response, a method that can employ up-to-the-minute relevant information. We train and evaluate such models on a newly collected dataset of human-human conversa- tions whereby one of the speakers is given ac- cess to internet search during knowledge-driven discussions in order to ground their responses. We find that search-query based access of the internet in conversation provides superior per- formance compared to existing approaches that either use no augmentation or FAISS-based re- trieval (Lewis et al., 2020b). 1 Introduction Open-domain dialogue, which involves chat about any topic, rather than a specific goal-directed topic, is commonly studied by training large language models (Adiwardana et al., 2020; Zhang et al., 2020; Roller et al., 2021). These models are trained either in a encoder-decoder or decoder only set- ting on large datasets of human-human conversa- tions, and any knowledge obtained during training is stored in the weights of the model. Such static language modeling fails to take into account the dynamic state of the world, where new information is coming in by the day – or even by the minute – as the knowledge in static models is gleaned from the point in time when the dataset was collected, and then frozen into the model that is trained; see Figure 1: Cherry picked example of a model with internet-augmentation trained on our new Wizard of the Internet task. The model is able to correctly suggest a pizza place in Princeton, complete with its address and phone number, by searching the internet. (Lazaridou et al., 2021) for criticisms of this ap- proach. Further, static language models are known to hallucinate, that is they generate plausible look- ing statements that are factually incorrect, which can be interpreted as a form of lossy compression when employing training to encode that knowledge within the weights of a neural network; see Shuster et al. (2021) for an in-depth study. In this work we study generative models that are instead capable of accessing the vast knowl- edge of the internet dynamically in order to inform their responses. Utilizing encoder-decoder archi- 8460 tectures, we consider models …

Content truncated to 500 words