Lexical Knowledge Internalization for Neural Dialog Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7945 - 7958 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Lexical Knowledge Internalization for Neural Dialog Generation Zhiyong Wu1,2 ∗, Wei Bi3 †, Xiang Li4, Lingpeng Kong1,2, Ben Kao1 1The University of Hong Kong, 3Tencent AI Lab, 2Shanghai AI Lab, 4East China Normal University 1{zywu,lpk,kao}@cs.hku.hk, 3victoriabi@tencent.com, 4xiangli@dase.ecnu.edu.cn Abstract We propose knowledge internalization (KI), which aims to complement the lexical knowl- edge into neural dialog models. Instead of further conditioning the knowledge-grounded dialog (KGD) models on externally retrieved knowledge, we seek to integrate knowledge about each input token internally into the model’s parameters. To tackle the challenge due to the large scale of lexical knowledge, we adopt the contrastive learning approach and create an effective token-level lexical knowl- edge retriever that requires only weak supervi- sion mined from Wikipedia. We demonstrate the effectiveness and general applicability of our approach on various datasets and diversi- fied model structures. 1 Introduction Vacuous responses (Li et al., 2016; Ghazvinine- jad et al., 2018), such as, I don’t know, are com- monly observed in end-to-end neural dialog mod- els (Shang et al., 2015; Sordoni et al., 2015). This is mostly because these models ignore the knowledge that resides in people’s minds during a conversation. To bridge the gap, many existing works (Moghe et al., 2018; Dinan et al., 2018) have attempted to condition the dialog model on external knowledge, either a sentence or a paragraph, retrieved based on the utterance and/or previous context. This curates datasets with utterance-response-knowledge triples (see Fig 1(a)). These knowledge-grounded dialog (KGD) models, despite demonstrated effectiveness, suffer from two major problems. First, equipping models with sentence-level knowledge alone will limit responses’ informative- ness and diversity. As shown in Fig 1(a), with the knowledge retrieved giving the utterance, a KGD model can relate J.K Rowling to Khalsa Aid. How- ever, retrieval based solely on sentence embeddings ∗The majority of this work was done while the first author was interning at Tencent AI Lab. †Corresponding author J.K Rowling donates a lot of money for COVID-19 relief work in India. Yes. Khalsa Aid received the donation. JK Rowling donated a six-figure amount to Khalsa Aid for their COVID-19 relief work in India. Retrieve (a) An utterance-response-knowledge triple. J.K Rowling is a British author and philanthropist. Really? J.K Rowling is my favorite British writer. I now respect her even more. Donate is to give (money or goods) for a good cause, for example to a charity. Wow. It’s so good of her. COVID-19, also known as the coronavirus, is a contagious disease caused severe acute …. Hope this will help stop the coronavirus in India. J.K Rowling: Donate: COVID-19: (b) Responses based on different lexical knowledge. Figure 1: (a) An exemplary KGD data sample with an utterance (top), a response (bottom), and a sentence- level knowledge (middle). (b) A list of lexical knowl- edge (in grey rectangle) related to words from the utter- ance in (a), and …

Content truncated to 500 words