Conditional Generation and Snapshot Learning in Neural Dialogue Systems

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2153–2162, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Conditional Generation and Snapshot Learning in Neural Dialogue Systems Tsung-Hsien Wen, Milica Gaˇsi´c, Nikola Mrkˇsi´c, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, Steve Young Cambridge University Engineering Department, Trumpington Street, Cambridge, CB2 1PZ, UK {thw28,mg436,nm480,lmr46,phs26,su259,djv27,sjy11}@cam.ac.uk Abstract Recently a variety of LSTM-based condi- tional language models (LM) have been ap- plied across a range of language generation tasks. In this work we study various model ar- chitectures and different ways to represent and aggregate the source information in an end- to-end neural dialogue system framework. A method called snapshot learning is also pro- posed to facilitate learning from supervised sequential signals by applying a companion cross-entropy objective function to the condi- tioning vector. The experimental and analyt- ical results demonstrate firstly that competi- tion occurs between the conditioning vector and the LM, and the differing architectures provide different trade-offs between the two. Secondly, the discriminative power and trans- parency of the conditioning vector is key to providing both model interpretability and bet- ter performance. Thirdly, snapshot learning leads to consistent performance improvements independent of which architecture is used. 1 Introduction Recurrent Neural Network (RNN)-based condi- tional language models (LM) have been shown to be very effective in tackling a number of real world problems, such as machine translation (MT) (Cho et al., 2014) and image caption generation (Karpa- thy and Fei-Fei, 2015). Recently, RNNs were ap- plied to task of generating sentences from an ex- plicit semantic representation (Wen et al., 2015a). Attention-based methods (Mei et al., 2016) and Long Short-term Memory (LSTM)-like (Hochreiter and Schmidhuber, 1997) gating mechanisms (Wen et al., 2015b) have both been studied to improve gen- eration quality. Although it is now clear that LSTM- based conditional LMs can generate plausible nat- ural language, less effort has been put in compar- ing the different model architectures. Furthermore, conditional generation models are typically tested on relatively straightforward tasks conditioned on a single source (e.g. a sentence or an image) and where the goal is to optimise a single metric (e.g. BLEU). In this work, we study the use of condi- tional LSTMs in the generation component of neu- ral network (NN)-based dialogue systems which de- pend on multiple conditioning sources and optimis- ing multiple metrics. Neural conversational agents (Vinyals and Le, 2015; Shang et al., 2015) are direct extensions of the sequence-to-sequence model (Sutskever et al., 2014) in which a conversation is cast as a source to target transduction problem. However, these mod- els are still far from real world applications be- cause they lack any capability for supporting domain specific tasks, for example, being able to interact with databases (Sukhbaatar et al., 2015; Yin et al., 2016) and aggregate useful information into their re- sponses. Recent work by Wen et al. (2016a), how- ever, proposed an end-to-end trainable neural dia- logue system that can assist users to complete spe- cific tasks. …

Content truncated to 500 words