Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 100–110, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 100 Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog Ryuichi Takanobu1, Hanlin Zhu2, Minlie Huang1∗ Institute for AI, BNRist, 1DCST, 2IIIS, Tsinghua University, Beijing, China gxly19@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn Abstract Dialog policy decides what and how a task- oriented dialog system will respond, and plays a vital role in delivering effective conversa- tions. Many studies apply Reinforcement Learning to learn a dialog policy with the re- ward function which requires elaborate de- sign and pre-specified user goals. With the growing needs to handle complex goals across multiple domains, such manually designed re- ward functions are not affordable to deal with the complexity of real-world tasks. To this end, we propose Guided Dialog Policy Learn- ing, a novel algorithm based on Adversarial Inverse Reinforcement Learning for joint re- ward estimation and policy optimization in multi-domain task-oriented dialog. The pro- posed approach estimates the reward signal and infers the user goal in the dialog sessions. The reward estimator evaluates the state-action pairs so that it can guide the dialog policy at each dialog turn. Extensive experiments on a multi-domain dialog dataset show that the dia- log policy guided by the learned reward func- tion achieves remarkably higher task success than state-of-the-art baselines. 1 Introduction Dialog policy, deciding the next action that the dialog agent should take at each turn, is a cru- cial component of a task-oriented dialog system. Among many models, Reinforcement Learning (RL) is commonly used to learn dialog policy (Fatemi et al., 2016; Peng et al., 2017; Chen et al., 2017; Yarats and Lewis, 2018; Lei et al., 2018; He et al., 2018; Su et al., 2018), where users are mod- eled as a part of the environment and the policy is learned through interactions with users. While it is too expensive to learn directly from real users since RL requires a large number of ∗Corresponding author U: I’m looking for a hotel to stay that has 5 stars and cheap price range. S: I am sorry that there is no such hotel, would you like to reserve a 3-star hotel as an alternative? U: I’d prefer a 4-star hotel even if it’s a bit expensive. Oh, and I need parking. S: OK, I find a moderately priced 4-star hotel that includes parking and free wifi. U: Are there any places to eat around it? S: Many. Japanese, Indian, French, etc. What kind of food would you like? Table 1: An example of the multi-domain task-oriented dialog between the user (U) and the system (S). The di- alog proceeds successfully because the system informs the user that no matching hotel exists (the first turn), identifies the new user goal about parking (the second turn), and shifts the topic to the restaurant domain (the third turn), which well understands the user’s demand. samples to train, …

Content truncated to 500 words