Knowledge-Grounded Dialogue Generation with Term-level De-noising

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2972–2983 August 1–6, 2021. ©2021 Association for Computational Linguistics 2972 Knowledge-Grounded Dialogue Generation with Term-level De-noising Wen Zheng1, Natasa Milic-Frayling1, Ke Zhou1,2 1University of Nottingham 2Nokia Bell Labs {wen.zheng,ke.zhou}@nottingham.ac.uk natasa-milic@frayling.net Abstract Dialogue generation has been improved through injecting knowledge into generative models. However, addition of knowledge through simple selection of sentences or para- graphs is likely to introduce noise and dimin- ish the effectiveness of the generative models. In this paper, we present a novel Knowledge Term Weighting Model (KTWM) that in- corporates term-level de-noising of the se- lected knowledge. KTWM includes a mod- ule for generating Simulated Response Vec- tors (SRVs) and uses SRVs attention distri- butions with the knowledge embeddings to determine knowledge term weights. Our experiments demonstrate that KTWM, com- bined with various knowledge selection algo- rithms, consistently achieves statistically sig- nificant improvements over methods without term weighting when applied to two publicly available datasets Wizard of Wikipedia (Wiz) and Holl-E. The results are particularly im- proved for the Wiz test data with unseen topics, demonstrating the robustness of the KTWM noise-reduction approach. 1 Introduction Research in dialogue generation has rapidly evolved from sequence-to-sequence (Sutskever et al., 2014) and Transformer models (Vaswani et al., 2017) to approaches with pre-trained models such as BERT (Devlin et al., 2019), XLNet (Yang et al., 2019) and T5 (Raffel et al., 2020). More recently, it included techniques that use knowledge, in addition to the original posts, to improve the quality of the generated responses (Ghazvinine- jad et al. (2018), Moghe et al. (2018), Dinan et al. (2019), Galley et al. (2019), Lian et al. (2019), Zheng and Zhou (2019), Zhao et al. (2020a), Zhao et al. (2020b)).1 This approach is referred to as 1Previous works used a variety of terms to refer to a post such as ‘question’, ‘utterance’,‘source’ and ‘query’. Similarly Post: I am a big fan of education. I think people don’t realise how important it is. Ground-truth response: Sure, education is important since it facilitates learning and the acquisition of skills. Knowledge terms weighted by KTWM: Education is the process of facilitating learning , or the acquisition of knowledge , skills , values , beliefs , and habits Response generated by KTWM: I agree. Education is a great way to learn about facilitating learning. 0 0.2 0.4 0.6 0.8 1 Table 1: Example of a post, ground truth response, injected knowledge and generated response by the Knowledge Term Weighting Model (KTWM). The term highlights indicate the predicted probability of a term being useful. knowledge-grounded dialogue generation and is the primary concern of this paper. In particular, we consider the key issue of effec- tively incorporating the selected knowledge into the generation process. For example, Weston et al. (2018) apply a retrieve and refine method to expand the post with the retrieved knowledge and then use it in the generation process. Lian et al. (2019) con- sider the post and response posterior distributions and the post prior distribution to …

Content truncated to 500 words