Dirichlet Latent Variable Hierarchical Recurrent Encoder-Decoder in Dialogue Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1267–1272, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1267 Dirichlet Latent Variable Hierarchical Recurrent Encoder-Decoder in Dialogue Generation Min Zeng Yisen Wang† Yuan Luo Department of Computer Science and Engineering Shanghai Jiao Tong University eewangyisen@gmail.com; {min zeng, yuanluo}@sjtu.edu.cn Abstract Variational encoder-decoders have achieved well-recognized performance in the dialogue generation task. Existing works simply as- sume the Gaussian priors of the latent vari- able, which are incapable of representing com- plex latent variables effectively. To address the issues, we propose to use the Dirich- let distribution with flexible structures to characterize the latent variables in place of the traditional Gaussian distribution, called Dirichlet Latent Variable Hierarchical Recur- rent Encoder-Decoder model (Dir-VHRED). Based on which, we further find that there is redundancy among the dimensions of latent variable, and the lengths and sentence patterns of the responses can be strongly correlated to each dimension of the latent variable. There- fore, controllable responses can be generated through specifying the value of each dimen- sion of the latent variable. Experimental re- sults on benchmarks show that our proposed Dir-VHRED yields substantial improvements on negative log-likelihood, word-embedding- based and human evaluations. 1 Introduction Recurrent neural networks (RNNs) (Bengio et al., 2003) have achieved great success on many nat- ural language processing tasks. For the dialogue generation task, a RNN-based Hierarchical Recur- rent Encoder-Decoder (HRED) was first proposed in (Serban et al., 2016). It consists of three RNNs: encoder RNN, context RNN and decoder RNN. Firstly, the encoder RNN encodes each utterance into a fixed-size real-valued vector through word embedding. Then, the last hidden state of en- coder RNN is feed into the context RNN to sum- marize the dialogue information. Finally, the de- coder RNN takes the last state of context RNN as †Corresponding author: Yisen Wang. input and produces the probability distribution of the word in the next utterance. Although it outperforms n-gram and other neu- ral network language models, HRED only pro- duces one word at a time, which is unable to fully grasp the holistic high-level syntactic prop- erties (e.g., topics, tones or sentiment) (Bowman et al., 2015). When the sentence grows longer, it has the drawback of tending to generate short and generic responses (Vinyals and Le, 2015). Thus, Serban et al. (2017) proposed the Variational Hi- erarchical Recurrent Encoder-Decoder (VHRED) by combining HRED with Variational Autoen- coders (VAEs) (Kingma and Welling, 2013) that introduced a latent variable to characterize the sentence-level representation for learning holistic properties. However, VHRED imposes a symmet- ric distribution (i.e., Gaussian distribution) to the latent variable, which, though facilitating analyz- ing, are incapable of representing complex latent variables effectively. Therefore, in order to gen- erate more meaningful and expressive responses, a more flexible and tractable prior distribution of the latent variable is needed. In this paper, we propose to use the Dirichlet distribution to characterize the latent variable in …

Content truncated to 500 words