Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3349–3358, Osaka, Japan, December 11-17 2016. Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation Lili Mou,1 Yiping Song,2 Rui Yan,3 Ge Li,1,∗ Lu Zhang,1 Zhi Jin1,∗ 1Key Laboratory of High Confidence Software Technologies (Peking University), MoE, China Institute of Software, Peking University, China ∗Corresponding authors 2Institute of Network Computing and Information Systems, Peking Univerity, China 3Institute of Computer Science and Technology, Peking University, China doublepower.mou@gmail.com {songyiping,ruiyan,lige,zhanglucs,zhijin}@pku.edu.cn Abstract Using neural networks to generate replies in human-computer dialogue systems is attracting in- creasing attention over the past few years. However, the performance is not satisfactory: the neural network tends to generate safe, universally relevant replies which carry little meaning. In this paper, we propose a content-introducing approach to neural network-based generative dia- logue systems. We first use pointwise mutual information (PMI) to predict a noun as a keyword, reflecting the main gist of the reply. We then propose seq2BF, a “sequence to backward and forward sequences” model, which generates a reply containing the given keyword. Experimental results show that our approach significantly outperforms traditional sequence-to-sequence mod- els in terms of human evaluation and the entropy measure, and that the predicted keyword can appear at an appropriate position in the reply. 1 Introduction Automatic human-computer conversation is a hot research topic in natural language processing (NLP). In past decades, researchers have developed various rule- or template-based systems, which are typically in vertical domains, e.g., transportation (Ferguson et al., 1996) and education (Graesser et al., 2005). In the open domain, data-driven approaches play an important role, because the diversity and uncertainty make it virtually impossible for humans to design rules or templates. Isbell et al. (2000) and Wang et al. (2013) use information retrieval methods to search for a reply from a pre-constructed database; Ritter et al. (2011) formalize conversation as a statistical machine translation task. Recently, the renewed prosperity of neural networks brings new opportunities to open-domain conver- sation (Vinyals and Le, 2015; Shang et al., 2015; Serban et al., 2016a; Li et al., 2016a). In these studies, researchers leverage sequence-to-sequence (seq2seq) models to encode a query (user-issued utterance) as a vector and to decode the vector into a reply. In both encoders and decoders, an RNN keeps one or a few hidden layers; at each time step, it reads a word and changes its state accordingly. RNNs are believed to be well capable of modeling word sequences, benefiting machine translation (Sutskever et al., 2014), abstractive summarization (Rush et al., 2015) and other tasks of natural language generation. Contrary to retrieval methods, neural network-based conversation systems are generative in that they can synthesize new utterances; results in the literature also show the superiority of seq2seq to phrase-based machine translation for dialogue systems (Shang et al., 2015). In our study, we focus on neural network-based generative short-text conversation, where we do not consider context information, following Wang et al. (2013) and Shang et al. (2015). …

Content truncated to 500 words