Injecting Domain Knowledge in Language Models for Task-oriented Dialogue Systems

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11962 - 11974 December 7-11, 2022 ©2022 Association for Computational Linguistics Injecting Domain Knowledge in Language Models for Task-Oriented Dialogue Systems ∗Denis Emelin1, Daniele Bonadiman2, ∗Sawsan Alqahtani3,4, Yi Zhang2, and Saab Mansour2 1University of Edinburgh , denis.emelin@gmail.com 2AWS AI Labs , {dbonadim,yizhngn,saabm}@amazon.com 3Princess Nourah Bint Abdulrahman , saalqhtani@pnu.edu.sa 4National Center of AI , sawalqahtani@nic.gov.sa Abstract Pre-trained language models (PLM) have ad- vanced the state-of-the-art across NLP applica- tions, but lack domain-specific knowledge that does not naturally occur in pre-training data. Previous studies augmented PLMs with sym- bolic knowledge for different downstream NLP tasks. However, knowledge bases (KBs) uti- lized in these studies are usually large-scale and static, in contrast to small, domain-specific, and modifiable knowledge bases that are prominent in real-world task-oriented dialogue (TOD) sys- tems. In this paper, we showcase the advan- tages of injecting domain-specific knowledge prior to fine-tuning on TOD tasks. To this end, we utilize light-weight adapters that can be eas- ily integrated with PLMs and serve as a repos- itory for facts learned from different KBs. To measure the efficacy of proposed knowledge injection methods, we introduce Knowledge Probing using Response Selection (KPRS) – a probe designed specifically for TOD models. Experiments1 on KPRS and the response gen- eration task show improvements of knowledge injection with adapters over strong baselines. 1 Introduction Pre-trained language models (PLMs), such as BERT (Devlin et al., 2018), BART (Lewis et al., 2020), GPT (Brown et al., 2020), and XLNet (Yang et al., 2019), have advanced the state-of-the-art of various natural language processing (NLP) tech- nologies and demonstrated an exceptional ability to store and utilize linguistic, factual, and com- monsense knowledge. Consequently, PLMs form the backbone of many recent NLP applications and have been successfully employed as modu- lar components in the context of task-oriented dia- logue (TOD), responsible for sub-tasks including ∗ Work performed while at AWS AI Labs 1https://github.com/amazon-research/ domain-knowledge-injection Figure 1: A high-level representation of the KB-adapter architecture (decoder only, for clarity). Adapter states are fused with the hidden states of the PLM to produce a knowledge-informed predictive distribution. Dashed elements are used only if multiple adapters are active. dialogue state tracking and response generation (Hosseini-Asl et al., 2020; Lee et al., 2021). Since they are exposed to large quantities of general data during training, PLMs store a wide variety of diverse and general knowledge in their parameters (Petroni et al., 2019) such as capitals of nations, biographical details of famous individuals, and other facts of varying granularity. Commer- cially deployed TOD systems, however, typically require access to more restricted, domain-specific categories of knowledge in order to produce in- formative and factually accurate responses to user queries.2 Such information may include addresses of particular local attractions, detailed restaurant menus, train routes, or ticket prices, and is unlikely to be found in the PLM’s training data. Due to its specialized nature, this knowledge is often stored in external knowledge bases (KBs) that are accessed at …

Content truncated to 500 words