HL-EncDec: A Hybrid-Level Encoder-Decoder for Neural Response Generation

Proceedings of the 27th International Conference on Computational Linguistics, pages 845–856 Santa Fe, New Mexico, USA, August 20-26, 2018. 845 HL-EncDec: A Hybrid-Level Encoder-Decoder for Neural Response Generation Sixing Wu∗, Dawei Zhang†, Ying Li ‡§, Xing Xie†, Zhonghai Wu‡ ∗School of Software and Microelectronics, Peking University, Beijing, China ‡National Research Center of Software Engineering, Peking University, Beijing, China †Microsoft Research Asia, Beijing, China §li.ying@pku.edu.cn Abstract Recent years have witnessed a surge of interest on response generation for neural conversation systems. Most existing models are implemented by following the Encoder-Decoder framework and operate sentences of conversations at word-level. The word-level model is suffering from the Unknown Words Issue and the Preference Issue, which seriously impact the quality of gen- erated responses, for example, generated responses may become irrelevant or too general (i.e. safe responses). To address these issues, this paper proposes a hybrid-level Encoder-Decoder model (HL-EncDec), which not only utilizes the word-level features but also character-level fea- tures. We conduct several experiments to evaluate HL-EncDec on a Chinese corpus, experimental results show our model significantly outperforms other non-word-level models in automatic met- rics and human annotations and is able to generate more informative responses. We also conduct experiments with a small-scale English dataset to show the generalization ability. 1 Introduction Nowadays, conversation systems have gained a great progress due to the rapid development of big-data and deep learning techniques. A conversation system enables a computer to make human-like conver- sations with users. Massive conversational datasets can be easily accessed on the Web, which highly promotes both the academia and industry to turn to their attention to develop data-driven conversation systems (Vinyals and Le, 2015; Shang et al., 2015; Chen et al., 2017). A common approach to build a generation-based conversation system is to model it as a response generation task. And a response generation model is often learnt within the Encoder-Decoder framework from large-scale conversational data from the Web. The Encoder-Decoder is a state-of-the-art framework for SEQ2SEQ (sequence-to-sequence) tasks such as machine translation, abstractive summarization and response generation, etc.. Researchers have proposed several Encoder-Decoder based models, and most of them utilize Recurrent Neural Networks to build an Encoder and an attentional Decoder. Due to the fact that a word is the most basic unit in lin- guistics, and other larger language elements are built on words, the majority of models operate sentences at word-level. Namely, sentences should be segmented as sequences of words, thus an Encoder-Decoder model could read or generate a sentence with their own vocabularies. Since the computational complex- ity and the capacity limitation of internal high-speed storage such as GPU VRAM, vocabularies could only record finite words. However, there is an issue that the amount of total words is far more than the volume of vocabularies and the out-of-vocabulary words (denoted as OOVs) may appear in the con- versations. Meanwhile, another issue is that words already recorded in the vocabulary have different probabilities to be used in human’s dialogues, only a small part of high-frequency words may be …

Content truncated to 500 words