DocChat: An Information Retrieval Approach for Chatbot Engines Using Unstructured Documents

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 516–525, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics DocChat: An Information Retrieval Approach for Chatbot Engines Using Unstructured Documents Zhao Yan† ∗, Nan Duan‡ , Junwei Bao+ , Peng Chen§ , Ming Zhou‡ , Zhoujun Li† , Jianshe Zhou¶ †State Key Laboratory of Software Development Environment, Beihang University ‡Microsoft Research +Harbin Institute of Technology §Microsoft Search Technology Center ¶BAICIT, Capital Normal University †{yanzhao, lizj}@buaa.edu.cn +baojunwei001@gmail.com ‡§{nanduan, peche, mingzhou}@microsoft.com ¶zhoujs@cnu.edu.cn Abstract Most current chatbot engines are designed to reply to user utterances based on exist- ing utterance-response (or Q-R)1 pairs. In this paper, we present DocChat, a novel information retrieval approach for chat- bot engines that can leverage unstructured documents, instead of Q-R pairs, to re- spond to utterances. A learning to rank model with features designed at different levels of granularity is proposed to mea- sure the relevance between utterances and responses directly. We evaluate our pro- posed approach in both English and Chi- nese: (i) For English, we evaluate Doc- Chat on WikiQA and QASent, two answer sentence selection tasks, and compare it with state-of-the-art methods. Reasonable improvements and good adaptability are observed. (ii) For Chinese, we compare DocChat with XiaoIce2, a famous chitchat engine in China, and side-by-side evalua- tion shows that DocChat is a perfect com- plement for chatbot engines using Q-R pairs as main source of responses. 1 Introduction Building chatbot engines that can interact with hu- mans with natural language is one of the most challenging problems in artificial intelligence. Along with the explosive growth of social media, like community question answering (CQA) web- sites (e.g., Yahoo Answers and WikiAnswers) and social media websites (e.g., Twitter and Weibo), ∗Contribution during internship at Microsoft Research. 1For convenience sake, we denote all utterance-response pairs (either QA pairs or conversational exchanges from so- cial media websites like Twitter) as Q-R pairs in this paper. 2http://www.msxiaoice.com the amount of utterance-response (or Q-R) pairs has experienced massive growth in recent years, and such a corpus greatly promotes the emergence of various data-driven chatbot approaches. Instead of multiple rounds of conversation, we only consider a much simplified task, short text conversation (STC) in which the response R is a short text and only depends on the last user utter- ance Q. Previous methods for the STC task mostly rely on Q-R pairs and fall into two categories: Retrieval-based methods (e.g., Ji et al., 2014). This type of methods first retrieve the most pos- sible ⟨ ˆQ, ˆR⟩ pair from a set of existing Q-R pairs, which best matches current utterance Q based on semantic matching models, then take ˆR as the re- sponse R. One disadvantage of such a method is that, for many specific domains, collecting such Q- R pairs is intractable. Generation based methods (e.g., Shang et al., 2015). This type of meth- ods usually uses an encoder-decoder framework which first encode Q as a vector representation, then feed this representation to decoder …

Content truncated to 500 words