A Collaborative Multi-agent Reinforcement Learning Framework for Dialog Action Decomposition

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7882–7889 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7882 A Collaborative Multi-agent Reinforcement Learning Framework for Dialog Action Decomposition Huimin Wang and Kam-Fai Wong The Chinese University of Hong Kong wanghm520@gmail.com kfwong@se.cuhk.edu.hk Abstract Most reinforcement learning methods for di- alog policy learning train a centralized agent that selects a predefined joint action concate- nating domain name, intent type, and slot name. The centralized dialog agent suffers from a great many user-agent interaction re- quirements due to the large action space. Be- sides, designing the concatenated actions is laborious to engineers and maybe struggled with edge cases. To solve these problems, we model the dialog policy learning problem with a novel multi-agent framework, in which each part of the action is led by a different agent. The framework reduces labor costs for action templates and decreases the size of the action space for each agent. Furthermore, we relieve the non-stationary problem caused by the changing dynamics of the environment as evolving of agents’ policies by introducing a joint optimization process that makes agents can exchange their policy information. Con- currently, an independent experience replay buffer mechanism is integrated to reduce the dependence between gradients of samples to improve training efficiency. The effectiveness of the proposed framework is demonstrated in a multi-domain environment with both user simulator evaluation and human evaluation. 1 Introduction Dialog policy optimization is one of the most crit- ical tasks of task-oriented dialog modeling. Re- cently, it has shown great potentials for using re- inforcement learning (RL) based methods to for- mulate dialog policy learning (Li et al., 2017; Peng et al., 2017). However, most of these methods learn a centralized agent based on the joint ac- tion space that covers predefined atomic action (Budzianowski et al., 2018), which is the concate- nation of domain name, intent type, and slot name, e.g. ‘restaurant-inform-address’, or both atomic actions and the top-k most frequent atomic action combinations(Lee et al., 2019a). The elaborate concatenated actions may achieve acceptable per- formance in simple cases, however, continuously suffer from being laborious to engineers and strug- gled with edge cases in multi-domain or complex scenes. Another drawback of the centralized agent is its exponential growth in the observation and ac- tions spaces with the growing number of domains (Lee et al., 2019b). To alleviate the problem of large user-agent in- teraction requirements caused by the large action space, a hierarchical reinforcement learning frame- work was proposed to learn the dialog policy that operates at different temporal scales (Peng et al., 2017). It has achieved promising results, however, is still up against some challenges. Firstly, their setting requires a rule-based critic to provide the intrinsic reward for the low-level agent. However, creating such a critic is not easy, especially in intri- cate scenarios. The man-made critic, somewhat inadvertently, may bias the convergent optimal. Moreover, the action space composed of intent and slot for the low-level agent can be still large, es- …

Content truncated to 500 words