Adversarial Learning for Neural Dialogue Generation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2157–2169 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Adversarial Learning for Neural Dialogue Generation Jiwei Li1, Will Monroe1, Tianlin Shi1, S´ebastien Jean2, Alan Ritter3 and Dan Jurafsky1 1Stanford University, Stanford, CA, USA 2New York University, NY, USA 3Ohio State University, OH, USA jiweil,wmonroe4,tianlins,jurafsky@stanford.edu sebastien@cs.nyu.edu ritter.1492@osu.edu Abstract In this paper, drawing intuition from the Turing test, we propose using adversar- ial training for open-domain dialogue gen- eration: the system is trained to pro- duce sequences that are indistinguish- able from human-generated dialogue ut- terances. We cast the task as a rein- forcement learning (RL) problem where we jointly train two systems, a generative model to produce response sequences, and a discriminator—analagous to the human evaluator in the Turing test— to distinguish between the human-generated dialogues and the machine-generated ones. The out- puts from the discriminator are then used as rewards for the generative model, push- ing the system to generate dialogues that mostly resemble human dialogues. In addition to adversarial training we de- scribe a model for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric, while avoid- ing a number of potential pitfalls. Ex- perimental results on several metrics, in- cluding adversarial evaluation, demonstrate that the adversarially-trained system gener- ates higher-quality responses than previous baselines. 1 Introduction Open domain dialogue generation (Ritter et al., 2011; Sordoni et al., 2015; Xu et al., 2016; Wen et al., 2016; Li et al., 2016b; Serban et al., 2016c, 2017) aims at generating meaningful and coher- ent dialogue responses given the dialogue history. Prior systems, e.g., phrase-based machine trans- lation systems (Ritter et al., 2011; Sordoni et al., 2015) or end-to-end neural systems (Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016a; Yao et al., 2015; Luan et al., 2016) approximate such a goal by predicting the next dialogue utterance given the dialogue history using the maximum likelihood estimation (MLE) objective. Despite its success, this over-simplified training objective leads to prob- lems: responses are dull, generic (Sordoni et al., 2015; Serban et al., 2016a; Li et al., 2016a), repeti- tive, and short-sighted (Li et al., 2016d). Solutions to these problems require answering a few fundamental questions: what are the cru- cial aspects that characterize an ideal conversation, how can we quantitatively measure them, and how can we incorporate them into a machine learning system? For example, Li et al. (2016d) manually define three ideal dialogue properties (ease of an- swering, informativeness and coherence) and use a reinforcement-learning framework to train the model to generate highly rewarded responses. Yu et al. (2016b) use keyword retrieval confidence as a reward. However, it is widely acknowledged that manually defined reward functions can’t possibly cover all crucial aspects and can lead to suboptimal generated utterances. A good dialogue model should generate utter- ances indistinguishable from human dialogues. Such a goal suggests a training objective resem- bling the idea of the Turing test (Turing, 1950). …

Content truncated to 500 words