A Discrete CVAE for Response Generation on Short-Text Conversation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1898–1908, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1898 A Discrete CVAE for Response Generation on Short-Text Conversation Jun Gao1∗, Wei Bi2†, Xiaojiang Liu2, Junhui Li1, Guodong Zhou1, Shuming Shi2 1School of Computer Science and Technology, Soochow University, Suzhou, China imgaojun@gmail.com, {lijunhui,gdzhou}@suda.edu.cn 2Tencent AI Lab, Shenzhen, China {victoriabi, kieranliu, shumingshi}@tencent.com Abstract Neural conversation models such as encoder- decoder models are easy to generate bland and generic responses. Some researchers propose to use the conditional variational autoencoder (CVAE) which maximizes the lower bound on the conditional log-likelihood on a continu- ous latent variable. With different sampled la- tent variables, the model is expected to gen- erate diverse responses. Although the CVAE- based models have shown tremendous poten- tial, their improvement of generating high- quality responses is still unsatisfactory. In this paper, we introduce a discrete latent variable with an explicit semantic meaning to improve the CVAE on short-text conversation. A major advantage of our model is that we can exploit the semantic distance between the latent vari- ables to maintain good diversity between the sampled latent variables. Accordingly, we pro- pose a two-stage sampling approach to enable efficient diverse variable selection from a large latent space assumed in the short-text conver- sation task. Experimental results indicate that our model outperforms various kinds of gener- ation models under both automatic and human evaluations and generates more diverse and in- formative responses. 1 Introduction Open-domain response generation (Perez-Marin, 2011; Sordoni et al., 2015) for single-round short text conversation (Shang et al., 2015), aims at generating a meaningful and interesting response given a query from human users. Neural gener- ation models are of growing interest in this topic due to their potential to leverage massive conver- sational datasets on the web. These generation models such as encoder-decoder models (Vinyals Lab.∗Work done when Jun Gao was interning at Tencent AI † Corresponding author and Le, 2015; Shang et al., 2015; Wen et al., 2015), directly build a mapping from the in- put query to its output response, which treats all query-response pairs uniformly and optimizes the maximum likelihood estimation (MLE). However, when the models converge, they tend to output bland and generic responses (Li et al., 2016a,c; Serban et al., 2016). Many enhanced encoder-decoder approaches have been proposed to improve the quality of gen- erated responses. They can be broadly classified into two categories (see Section 2 for details): (1) One that does not change the encoder-decoder framework itself. These approaches only change the decoding strategy, such as encouraging diverse tokens to be selected in beam search (Li et al., 2016a,b); or adding more components based on the encoder-decoder framework, such as the Gen- erative Adversarial Network (GAN)-based meth- ods (Xu et al., 2017; Zhang et al., 2018; Li et al., 2017) which add discriminators to perform adver- sarial training; (2) The second category modifies the encoder-decoder framework …

Content truncated to 500 words