Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 484–495 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1045 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 484–495 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1045 Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access Bhuwan Dhingra⋆ Lihong Li† Xiujun Li† Jianfeng Gao† Yun-Nung Chen‡ Faisal Ahmed† Li Deng† ⋆Carnegie Mellon University, Pittsburgh, PA, USA †Microsoft Research, Redmond, WA, USA ‡National Taiwan University, Taipei, Taiwan ⋆bdhingra@andrew.cmu.edu †{lihongli,xiul,jfgao}@microsoft.com ‡y.v.chen@ieee.org Abstract This paper proposes KB-InfoBot1 — a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agents typ- ically need to interact with an external database to access real-world knowledge. Previous systems achieved this by issuing a symbolic query to the KB to retrieve en- tries based on their attributes. However, such symbolic operations break the differ- entiability of the system and prevent end- to-end training of neural dialogue agents. In this paper, we address this limitation by replacing symbolic queries with an in- duced “soft” posterior distribution over the KB that indicates which entities the user is interested in. Integrating the soft retrieval process with a reinforcement learner leads to higher task success rate and reward in both simulations and against real users. We also present a fully neural end-to-end agent, trained entirely from user feedback, and discuss its application towards person- alized dialogue agents. 1 Introduction The design of intelligent assistants which interact with users in natural language ranks high on the agenda of current NLP research. With an increas- ing focus on the use of statistical and machine learning based approaches (Young et al., 2013), the last few years have seen some truly remark- able conversational agents appear on the market (e.g. Apple Siri, Microsoft Cortana, Google Allo). These agents can perform simple tasks, answer 1The source code is available at: https://github. com/MiuLab/KB-InfoBot factual questions, and sometimes also aimlessly chit-chat with the user, but they still lag far be- hind a human assistant in terms of both the va- riety and complexity of tasks they can perform. In particular, they lack the ability to learn from interactions with a user in order to improve and adapt with time. Recently, Reinforcement Learn- ing (RL) has been explored to leverage user inter- actions to adapt various dialogue agents designed, respectively, for task completion (Gaˇsi´c et al., 2013), information access (Wen et al., 2016b), and chitchat (Li et al., 2016a). We focus on KB-InfoBots, a particular type of dialogue agent that helps users navigate a Knowl- edge Base (KB) in search of an entity, as illus- trated by the example in Figure 1. Such agents must necessarily query databases in order to re- trieve the requested information. This is usually done by performing semantic parsing on the input to construct a symbolic query representing the be- liefs of the agent about the user …

Content truncated to 500 words