Deep Reinforcement Learning for Dialogue Generation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Deep Reinforcement Learning for Dialogue Generation Jiwei Li1, Will Monroe1, Alan Ritter2, Michel Galley3, Jianfeng Gao3 and Dan Jurafsky1 1Stanford University, Stanford, CA, USA 2Ohio State University, OH, USA 3Microsoft Research, Redmond, WA, USA {jiweil,wmonroe4,jurafsky}@stanford.edu, ritter.1492@osu.edu {mgalley,jfgao}@microsoft.com Abstract Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be short- sighted, predicting utterances one at a time while ignoring their influence on future out- comes. Modeling the future direction of a di- alogue is crucial to generating coherent, inter- esting dialogues, a need which led traditional NLP models of dialogue to draw on reinforce- ment learning. In this paper, we show how to integrate these goals, applying deep reinforce- ment learning to model future reward in chat- bot dialogue. The model simulates dialogues between two virtual agents, using policy gradi- ent methods to reward sequences that display three useful conversational properties: infor- mativity, coherence, and ease of answering (re- lated to forward-looking function). We evalu- ate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conver- sation in dialogue simulation. This work marks a first step towards learning a neural conversa- tional model based on the long-term success of dialogues. 1 Introduction Neural response generation (Sordoni et al., 2015; Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016a; Wen et al., 2015; Yao et al., 2015; Luan et al., 2016; Xu et al., 2016; Wen et al., 2016; Li et al., 2016b; Su et al., 2016) is of growing inter- est. The LSTM sequence-to-sequence (SEQ2SEQ) model (Sutskever et al., 2014) is one type of neural generation model that maximizes the probability of generating a response given the previous dialogue turn. This approach enables the incorporation of rich context when mapping between consecutive dialogue turns (Sordoni et al., 2015) in a way not possible, for example, with MT-based dialogue models (Ritter et al., 2011). Despite the success of SEQ2SEQ models in di- alogue generation, two problems emerge: First, SEQ2SEQ models are trained by predicting the next dialogue turn in a given conversational context using the maximum-likelihood estimation (MLE) objective function. However, it is not clear how well MLE approximates the real-world goal of chatbot develop- ment: teaching a machine to converse with humans, while providing interesting, diverse, and informative feedback that keeps users engaged. One concrete example is that SEQ2SEQ models tend to generate highly generic responses such as “I don’t know” re- gardless of the input (Sordoni et al., 2015; Serban et al., 2016; Li et al., 2016a). This can be ascribed to the high frequency of generic responses found in the training set and their compatibility with a diverse range of conversational contexts. Yet “I don’t know” is apparently not a good action to take, since …

Content truncated to 500 words