Learning Goal-oriented Dialogue Policy with opposite Agent Awareness

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 122–132 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 122 Learning Goal-oriented Dialogue Policy with Opposite Agent Awareness Zheng Zhang†, Lizi Liao‡, Xiaoyan Zhu†, Tat-Seng Chua‡, Zitao Liu§, Yan Huang§, Minlie Huang†∗ † Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University ‡ School of Computing, National University of Singapore, Singpore § TAL Education Group †zhangz.goal@gmail.com {zxy-dcs,aihuang}@tsinghua.edu.cn ‡liaolizi.llz@gmail.com chuats@comp.nus.edu.sg §{liuzitao,galehuang}@100tal.com Abstract Most existing approaches for goal-oriented dialogue policy learning used reinforcement learning, which focuses on the target agent pol- icy and simply treats the opposite agent pol- icy as part of the environment. While in real- world scenarios, the behavior of an opposite agent often exhibits certain patterns or under- lies hidden policies, which can be inferred and utilized by the target agent to facilitate its own decision making. This strategy is common in human mental simulation by first imaging a specific action and the probable results be- fore really acting it. We therefore propose an opposite behavior aware framework for policy learning in goal-oriented dialogues. We esti- mate the opposite agent’s policy from its be- havior and use this estimation to improve the target agent by regarding it as part of the target policy. We evaluate our model on both coop- erative and competitive dialogue tasks, show- ing superior performance over state-of-the-art baselines. 1 Introduction In goal-oriented dialogue systems, dialogue pol- icy plays a crucial role by deciding the next action to take conditioned on the dialogue state. This problem is often formulated using reinforcement learning (RL) in which the user serves as the envi- ronment (Levin et al., 1997; Rieser and Lemon, 2011; Lemon and Pietquin, 2012; Young et al., 2013; Fatemi et al., 2016; Zhao and Eskenazi, 2016; Dhingra et al., 2016; Su et al., 2016; Li et al., 2017; Williams et al., 2017; Liu and Lane, 2017; Lip- ton et al., 2018; Liu et al., 2018; Gao et al., 2019; Takanobu et al., 2019, 2020; Jhunjhunwala et al., 2020). However, different from symbolic-based and simulation-based RL tasks, such as chess (Sil- ver et al., 2016) and video games (Mnih et al., ∗Corresponding author. 2015), which can get vast amounts of training in- teractions in low cost, dialogue systems require to learn directly from real users, which is too expen- sive. Therefore, there are some efforts using simu- lation methods to provide an affordable training environment. One principle direction for mitigat- ing this problem is to leverage human conversation data to build a user simulator, and then to learn the dialogue policy by making simulated interactions with the simulator (Schatzmann et al., 2006; Li et al., 2016; G¨ur et al., 2018). However, there always exist discrepancies be- tween simulated users and real users due to the inductive biases of …

Content truncated to 500 words