Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2231–2240 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning Baolin Peng? Xiujun Li† Lihong Li† Jianfeng Gao† Asli Celikyilmaz† Sungjin Lee† Kam-Fai Wong? †Microsoft Research, Redmond, WA, USA ?The Chinese University of Hong Kong, Hong Kong {blpeng, kfwong}@se.cuhk.edu.hk {xiul,lihongli,jfgao,aslicel,sule}@microsoft.com Abstract Building a dialogue agent to fulfill com- plex tasks, such as travel planning, is chal- lenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep rein- forcement learning approach to learning a dialogue manager that operates at differ- ent temporal scales. The dialogue man- ager consists of: (1) a top-level dialogue policy that selects among subtasks or op- tions, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our ap- proach leads to significant improvements over three baselines, two based on hand- crafted rules and the other based on flat deep reinforcement learning. 1 Introduction There is a growing demand for intelligent personal assistants, mainly in the form of dialogue agents, that can help users accomplish tasks ranging from meeting scheduling to vacation planning. How- ever, most of the popular agents in today’s market, such as Amazon Echo, Apple Siri, Google Home and Microsoft Cortana, can only handle very sim- ple tasks, such as reporting weather and requesting songs. Building a dialogue agent to fulfill complex tasks remains one of the most fundamental chal- lenges for the NLP community and AI in general. In this paper, we consider an important type of complex tasks, termed composite task, which con- sists of a set of subtasks that need to be fulfilled collectively. For example, in order to make a travel plan, we need to book air tickets, reserve a hotel, rent a car, etc. in a collective way so as to satisfy a set of cross-subtask constraints, which we call slot constraints. Examples of slot constraints for travel planning are: hotel check-in time should be later than the flight’s arrival time, hotel check-out time may be earlier than the return flight depart time, the number of flight tickets equals to that of hotel check-in people, and so on. It is common to learn a task-completion dia- logue agent using reinforcement learning (RL); see Su et al. (2016); Cuay´ahuitl (2017); Williams et al. (2017); Dhingra et al. (2017) and Li et al. (2017a) for a few recent examples. Compared to these dialogue agents developed for individ- ual domains, …

Content truncated to 500 words