Efficient Dialogue Complementary Policy Learning via Deep Q-network Policy and Episodic Memory Policy

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4311–4323 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4311 Efficient Dialogue Complementary Policy Learning via Deep Q-network Policy and Episodic Memory Policy Yangyang Zhao†‡, Zhenyu Wang†∗, Changxi Zhu‡ and Shihan Wang‡ †South China University of Technology ‡Utrecht University msyyz@mail.scut.edu.cn; wangzy@scut.edu.cn; cxzhu.cn@gmail.com; s.wang2@uu.nl Abstract Deep reinforcement learning has shown great potential in training dialogue policies. How- ever, its favorable performance comes at the cost of many rounds of interaction. Most of the existing dialogue policy methods rely on a sin- gle learning system, while the human brain has two specialized learning and memory systems, supporting to find good solutions without re- quiring copious examples. Inspired by the hu- man brain, this paper proposes a novel com- plementary policy learning (CPL) framework, which exploits the complementary advantages of the episodic memory (EM) policy and the deep Q-network (DQN) policy to achieve fast and effective dialogue policy learning. In or- der to coordinate between the two policies, we proposed a confidence controller to con- trol the complementary time according to their relative efficacy at different stages. Further- more, memory connectivity and time prun- ing are proposed to guarantee the flexible and adaptive generalization of the EM policy in di- alog tasks. Experimental results on three di- alogue datasets show that our method signif- icantly outperforms existing methods relying on a single learning system. 1 Introduction Dialogue policy, one of the most critical modules of task-oriented dialogue systems, aims to determine system responses based on current states (Zhang et al., 2019a). One of the earliest methods is the rule-based policy (Litman and Allen, 1987; Bos et al., 2003). Although this method often has ac- ceptable performance, handcrafting rules are ex- pensive and non-extensible. Recently, deep rein- forcement learning (RL) has become a mainstream method for training dialogue policies (Cuayáhuitl, 2016; Li et al., 2017; Peng et al., 2017, 2018). Since Deep RL-based methods are learning in an online fashion, a large amount of interaction with ∗Corresponding author real users is required, which is generally infeasible in practical application (Fatemi et al., 2016; Dhin- gra et al., 2017; Su et al., 2018; Wu et al., 2019). Intuitively, the reward-based learning mech- anism in Deep RL (DRL) coincides with the dopamine-centered regulation in the human brain (O’Reilly et al., 2014). Human brains have two differentially specialized learning and memory sys- tems for collaboration, allowing them to find good solutions without requiring copious examples (Mc- Clelland et al., 1995; Norman and O"Reilly, 2003; O’Reilly et al., 2014). However, most of the DRL- based dialogue policies (Chen et al., 2017b; Peng et al., 2018; Lipton et al., 2018; Takanobu et al., 2019; Wu et al., 2019; Wang et al., 2020) rely on a single learning system, which neglects the hu- man brain’s memory structure. Consequently, we imitate the human brain to construct an efficient complementary policy learning (CPL) model with learning and memory systems for cooperation. Inspired by cognitive neuroscience studies (Sutherland and Rudy, …

Content truncated to 500 words