Incremental Transformer with Deliberation Decoder for Document Grounded Conversations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 12–21 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 12 Incremental Transformer with Deliberation Decoder for Document Grounded Conversations Zekang Li†♦, Cheng Niu‡, Fandong Meng‡∗, Yang Feng♦, Qian Li♠, Jie Zhou‡ †Dian Group, School of Electronic Information and Communications Huazhong University of Science and Technology ‡Pattern Recognition Center, WeChat AI, Tencent Inc, China ♦Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences ♠School of Computer Science and Engineering, Northeastern University, China zekangli97@gmail.com, {chengniu,fandongmeng,jiezhou}@tencent.com fengyang@ict.ac.cn, qianli@stumail.neu.edu.cn Abstract Document Grounded Conversations is a task to generate dialogue responses when chatting about the content of a given document. Ob- viously, document knowledge plays a critical role in Document Grounded Conversations, while existing dialogue models do not exploit this kind of knowledge effectively enough. In this paper, we propose a novel Transformer- based architecture for multi-turn document grounded conversations. In particular, we de- vise an Incremental Transformer to encode multi-turn utterances along with knowledge in related documents. Motivated by the hu- man cognitive process, we design a two-pass decoder (Deliberation Decoder) to improve context coherence and knowledge correctness. Our empirical study on a real-world Document Grounded Dataset proves that responses gen- erated by our model significantly outperform competitive baselines on both context coher- ence and knowledge relevance. 1 Introduction Past few years have witnessed the rapid develop- ment of dialogue systems. Based on the sequence- to-sequence framework (Sutskever et al., 2014), most models are trained in an end-to-end man- ner with large corpora of human-to-human di- alogues and have obtained impressive success (Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016; Serban et al., 2016). While there is still a long way for reaching the ultimate goal of di- alogue systems, which is to be able to talk like humans. And one of the essential intelligence to achieve this goal is the ability to make use of knowledge. ∗ Fandong Meng is the corresponding author of the pa- per. This work was done when Zekang Li was interning at Pattern Recognition Center, WeChat AI, Tencent. There are several works on dialogue sys- tems exploiting knowledge. The Mem2Seq (Madotto et al., 2018) incorporates structured knowledge into the end-to-end task-oriented di- alogue. Liu et al. (2018) introduces fact- matching and knowledge-diffusion to generate meaningful, diverse and natural responses using structured knowledge triplets. Ghazvininejad et al. (2018), Parthasarathi and Pineau (2018), Yavuz et al. (2018), Dinan et al. (2018) and Lo and Chen (2019) apply unstructured text facts in open-domain dialogue systems. These works mainly focus on integrating factoid knowledge into dialogue systems, while factoid knowledge requires a lot of work to build up, and is only limited to expressing precise facts. Documents as a knowledge source provide a wide spectrum of knowledge, including but not limited to factoid, event updates, subjective opinion, etc. Recently, intensive research has been applied on using documents as knowledge sources for Question- Answering (Chen et …

Content truncated to 500 words