TSDG: Content-aware Neural Response Generation with Two-stage Decoding Process

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2121–2126 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2121 TSDG: Content-aware Neural Response Generation with Two-stage Decoding Process Junsheng Kong∗, Zhicheng Zhong∗, Yi Cai†, Xin Wu and Da Ren School of Software Engineering, South China University of Technology, Guangzhou, China Key Laboratory of Big Data and Intelligent Robot (South China University of Technology), Ministry of Education sescut kongjunsheng@mail.scut.edu.cn, ycai@scut.edu.cn Abstract Neural response generative models have achieved remarkable progress in recent years but tend to yield irrelevant and uninformative responses. One of the reasons is that encoder- decoder based models always use a single de- coder to generate a complete response at a stroke. This tends to generate high-frequency function words with less semantic information rather than low-frequency content words with more semantic information. To address this is- sue, we propose a content-aware model with two-stage decoding process named Two-stage Dialogue Generation (TSDG). We separate the decoding process of content words and func- tion words so that content words can be gen- erated independently without the interference of function words. Experimental results on two datasets indicate that our model signifi- cantly outperforms several competitive gener- ative models in terms of automatic evaluation and human evaluation. 1 Introduction With the development of deep learning, the open- domain neural response generation has achieved remarkable progress (Li et al., 2016; Serban et al., 2017b; Chen et al., 2019) in resent years. At present, most of generative models are based on encoder-decoder framework (Cho et al., 2014; Shang et al., 2015). In the decoding process, these models always use a single decoder to generate the final response at a stroke in a left-to-right man- ner. However, we find it hard for these methods to model the dependency of semantic between post and response which causes irrelevant and uninfor- mative responses. We analyze this problem from the perspective of linguistics as following. In linguistics, there are two different types of words to form a sentence, namely content words ∗ Both of authors contributed equally to this research. † Corresponding author Figure 1: An example of content-aware response gen- eration. (words which have substantive lexical content) and function words (words which essentially serve to make grammatical properties) (Hill, 1952). For the response “I am going to read an interesting book.” in Figure 1, content words “read, interesting, book” give us the most important semantic information which establishes the semantic dependency with the post, while function words “I, am, going, to, an” are used to stitch content words together. High- quality content words are a critical component of a relevant and informative response. Although func- tion words are small in numbers (less than 0.04% of our vocabulary), they account for over half of the words used in our daily speech (Rochon et al., 2000). Therefore, function words are always high- frequency relative to the content words. In vanilla encoder-decoder models, these models always use a single decoder to generate a complete response at a stroke. …

Content truncated to 500 words