Task-Completion Dialogue Policy Learning via Monte Carlo Tree Search with Dueling Network

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3461–3471, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3461 Task-Completion Dialogue Policy Learning via Monte Carlo Tree Search with Dueling Network Sihan Wang, Kaijie Zhou∗, Kunfeng Lai, Jianping Shen Ping An Life Insurance of China, Ltd. {wangsihan088, zhoukaijie002}@pingan.com.cn {laikunfeng597, shenjianping324}@pingan.com.cn Abstract We introduce a framework of Monte Carlo Tree Search with Double-q Dueling network (MCTS-DDU) for task-completion dialogue policy learning. Different from the previ- ous deep model-based reinforcement learning methods, which uses background planning and may suffer from low-quality simulated expe- riences, MCTS-DDU performs decision-time planning based on dialogue state search trees built by Monte Carlo simulations and is robust to the simulation errors. Such idea arises nat- urally in human behaviors, e.g. predicting oth- ers’ responses and then deciding our own ac- tions. In the simulated movie-ticket booking task, our method outperforms the background planning approaches significantly. We demon- strate the effectiveness of MCTS and the du- eling network in detailed ablation studies, and also compare the performance upper bounds of these two planning methods. 1 Introduction Designing a task-completion dialogue system has become an important task due to its huge com- mercial values. The dialogue agent aims to help users to complete a single or multi-domain task, e.g. booking a flight and making a hotel reservation. The core of such a system is the dialogue policy module, which enables the agent to respond prop- erly and provide users with the desired information. Early work has shown that dialogue policy learn- ing can be designed as a Markov Decision Process (MDP) (Singh et al., 2002; He et al., 2018; Zhao et al., 2019; Takanobu et al., 2019). Reinforcement learning (RL) is a common framework to solve MDP but it requires huge amounts of interactions with real users, which is generally infeasible in the real world. One way to work around this problem ∗Corresponding author is by designing a user simulator using the real hu- man conversation data (Schatzmann et al., 2007; Li et al., 2016). Recently, following these ideas, many re- searchers have applied deep model-based RL meth- ods to task-completion dialogue policy learning (Peng et al., 2018; Su et al., 2018; Wu et al., 2018). In a model-based method, the agent not only up- dates its action value or policy function through real experiences but also learns how the environ- ment produces the next states and rewards. The learned environment is called a model, which can be further used to generate simulated experiences. Using both real and simulated experiences is re- ferred to as background planning and can substan- tially improve the learning efficiency (Sutton and Barto, 2018). Peng et al. (2018) extend Dyna-Q (Sutton, 1990; Sutton et al., 2012) to Deep Dyna-Q (DDQ) for dialogue policy learning and achieve appealing results. However, since model learning cannot be perfect, some simulated experiences with large errors may hinder policy learning (Su et al., 2018). Su et al. (2018) propose to train a discrim- inator to …

Content truncated to 500 words