A Neural Network Approach to Context-Sensitive Generation of Conversational Responses

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 196–205, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics A Neural Network Approach to Context-Sensitive Generation of Conversational Responses Alessandro Sordoni1∗† Michel Galley2† Michael Auli3∗ Chris Brockett2 Yangfeng Ji4∗ Margaret Mitchell2 Jian-Yun Nie1∗ Jianfeng Gao2 Bill Dolan2 1DIRO, Universit´e de Montr´eal, Montr´eal, QC, Canada 2Microsoft Research, Redmond, WA, USA 3Facebook AI Research, Menlo Park, CA, USA 4Georgia Institute of Technology, Atlanta, GA, USA Abstract We present a novel response generation sys- tem that can be trained end to end on large quantities of unstructured Twitter conversa- tions. A neural network architecture is used to address sparsity issues that arise when in- tegrating contextual information into classic statistical models, allowing the system to take into account previous dialog utterances. Our dynamic-context generative models show con- sistent gains over both context-sensitive and non-context-sensitive Machine Translation and Information Retrieval baselines. 1 Introduction Until recently, the goal of training open-domain con- versational systems that emulate human conversation has seemed elusive. However, the vast quantities of conversational exchanges now available on so- cial media websites such as Twitter and Reddit raise the prospect of building data-driven models that can begin to communicate conversationally. The work of Ritter et al. (2011), for example, demonstrates that a response generation system can be constructed from Twitter conversations using statistical machine trans- lation techniques, where a status post by a Twitter user is “translated” into a plausible looking response. However, an approach such as that presented in Rit- ter et al. (2011) does not address the challenge of *The entirety of this work was conducted while at Microsoft Research. †Corresponding authors: Alessandro Sordoni (sor- donia@iro.umontreal.ca) and Michel Galley (mgal- ley@microsoft.com). context because of your game ? message yeah i’m on my way now response ok good luck ! Figure 1: Example of three consecutive utterances occur- ring between two Twitter users A and B. generating responses that are sensitive to the context of the conversation. Broadly speaking, context may be linguistic or involve grounding in the physical or virtual world, but we here focus on linguistic context. The ability to take into account previous utterances is key to building dialog systems that can keep con- versations active and engaging. Figure 1 illustrates a typical Twitter dialog where the contextual infor- mation is crucial: the phrase “good luck” is plainly motivated by the reference to “your game” in the first utterance. In the MT model, such contextual sensitiv- ity is difficult to capture; moreover, naive injection of context information would entail unmanageable growth of the phrase table at the cost of increased sparsity, and skew towards rarely-seen context pairs. In most statistical approaches to machine translation, phrase pairs do not share statistical weights regard- less of their intrinsic semantic commonality. We propose to address the challenge of context- sensitive response generation by using continuous representations or embeddings of words and phrases to compactly encode semantic and syntactic simi- larity. …

Content truncated to 500 words