External Knowledge Acquisition for End-to-End Document-Oriented Dialog Systems

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3633–3647 May 2-6, 2023 ©2023 Association for Computational Linguistics External Knowledge Acquisition for End-to-End Document-Oriented Dialog Systems Tuan M. Lai2∗, Giuseppe Castellucci1, Saar Kuzi1, Heng Ji1, Oleg Rokhlenko1 1Amazon 2University of Illinois Urbana-Champaign {giusecas, skuzi, jihj, olegro}@amazon.com tuanml2@illinois.edu Abstract End-to-end neural models for conversational AI often assume that a response can be gen- erated by considering only the knowledge ac- quired by the model during training. Document- oriented conversational models make a simi- lar assumption by conditioning the input on the document and assuming that any other knowledge is captured in the model’s weights. However, a conversation may refer to exter- nal knowledge sources. In this work, we present EKo-DoC, an architecture for document- oriented conversations with access to external knowledge: we assume that a conversation is centered around a topic document and that external knowledge is needed to produce re- sponses. EKo-DoC includes a dense passage retriever, a re-ranker, and a response generation model. We train the model end-to-end by us- ing silver labels for the retrieval and re-ranking components that we automatically acquire from the attention signals of the response genera- tion model. We demonstrate with automatic and human evaluations that incorporating exter- nal knowledge improves response generation in document-oriented conversations. Our architec- ture achieves new state-of-the-art results on the Wizard of Wikipedia dataset, outperforming a competitive baseline by 10.3% in Recall@1 and 7.4% in ROUGE-L. 1 Introduction Recent advances in Natural Language Processing (NLP) allowed us to build complex conversational systems for various scenarios, such as task-oriented (Radlinski and Craswell, 2017; Wen et al., 2017), chit-chat (Khatri et al., 2018; Zhou et al., 2020), and even for guiding users in performing complex real-world tasks (Gottardi et al., 2022). When using deep neural networks for implementing a conver- sational agent, a common practice is to use many historical conversations to train the model to pro- ∗* Work done during internship at Amazon. duce responses that are related to a dialog con- text. In this setting, a response is generated solely from the knowledge acquired during the training of the network (Vinyals and Le, 2015), and no other knowledge sources are used at inference time. A more effective way of producing dialog re- sponses would be to incorporate external knowl- edge into the model. This is, for example, the case of systems that make use of document/passage re- trieval in QA (Lewis et al., 2020b). In this work, we consider the setting in which a conversation is grounded in a target topic but also in external knowledge, in the form of documents. For exam- ple, Figure 1 shows a conversation with respect to a target topic (i.e., San Diego Comic-Con) repre- sented by a topic document. During the conver- sation, some turns may refer to other documents from an external Knowledge Base (KB) providing additional information (e.g., Shel Dorf). In our preliminary studies, we estimated that about 36% of the dialogs in the …

Content truncated to 500 words