Gaussian Process based Deep Dyna-Q approach for Dialogue Policy Learning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1786–1795 August 1–6, 2021. ©2021 Association for Computational Linguistics 1786 Gaussian Process based Deep Dyna-Q Approach for Dialogue Policy Learning Guanlin Wu 1,2,∗ Wenqi Fang 3,∗,† Ji Wang 1,∗ Jiang Cao 2 Weidong Bao 1 Yang Ping 2 Xiaomin Zhu 1 Zheng Wang 4 1National University of Defense Technology, Changsha, China 2Academy of Military Science, Beijing, China 3Nanhu Laboratory, Jiaxing, China 4Shenzhen Institutes of Advanced Technology, CAS, Shenzhen, China {wuguanlin16,wangji,wdbao,xmzhu}@nudt.edu.cn ocean.py@163.com wqfang@nanhulab.ac.cnamscaojiang@126.com zheng.wang@siat.ac.cn Abstract Applying reinforcement learning to dialogue policy learning requires prohibitively large rounds of human-machine interactions. To improve the learning performance, the Deep Dyna-Q framework with a world model that imitates real users is widely used in recent years. Unfortunately, how to build an effec- tive world model and how to evaluate the ex- periences generated by the world model effi- ciently have not been well studied. In order to further improve the effectiveness and effi- ciency of dialogue policy learning, we present a novel Gaussian Process based Deep Dyna-Q approach in this paper. The Gaussian Process model, which is analytically tractable and fits for small-sample problems, is introduced to build the world model. In addition, we design a highly efficient Kullback-Leibler divergence based discriminator to evaluate the quality of experiences generated by the world model. Ex- tensive experiments validate the effectiveness and robustness of our proposed approach. The task-completion success rate can be improved by about 20% with fewer human-machine in- teractions. 1 Introduction Task-completion dialogue policy learning aims to build a task-completion dialogue system that can help people complete a specific single task or multi- domain tasks through several rounds of natural lan- guage interactions. It has been widely used in chat robots and personal voice assistants, such as Siri of Apple and Cortana of Microsoft. Reinforcement learning (RL) becomes the main- stream dialogue policy learning method in recent years (Chen et al., 2020; Saha et al., 2020; Li et al., 2020). Based on the RL, the task-completion dia- logue system can gradually adjust policy through ∗The three authors contribute equally to this work †Corresponding author: wqfang@nanhulab.ac.cn interacting with real users to improve performance. However, the vanilla RL methods require many rounds of human-machine dialogue interactions be- fore getting a satisfactory dialogue policy, which not only increases the training cost but also dete- riorates user experience during the early training phase. In order to address the above problem and accel- erate the learning process of dialogue policy, Deep Dyna-Q (DDQ) (Peng et al., 2018) is proposed based on the Dyna-Q framework where a environ- ment model, known as world model, is introduced to generate simulated user experiences in the dy- namic environment. The world model is trained by the real user experience to make itself act more like real users. During the dialogue policy learning, the dialogue agent is trained by both real experiences collected from interacting with real users and sim- ulated experiences collected from interacting with the world model. By introducing the world model, …

Content truncated to 500 words