Scheduled Dialog Policy Learning: An Automatic Curriculum Learning Framework for Task-oriented Dialog System

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1091–1102 August 1–6, 2021. ©2021 Association for Computational Linguistics 1091 Scheduled Dialog Policy Learning: An Automatic Curriculum Learning Framework for Task-oriented Dialog System Sihong Liu1∗, Jinchao Zhang2,Keqing He1, Weiran Xu1†, Jie Zhou2 1 Beijing University of Posts and Telecommunications, China 2 Pattern Recognition Center, WeChat AI, Tencent Inc, China {liusihong,kqin,xuweiran}@bupt.edu.cn {dayerzhang,withtomzhou}@tencent.com Abstract In reinforcement learning (RL) based task- oriented dialogue systems, users act as the en- vironment and the agent learns the policy by interacting with users. However, due to the subjectivity of different users, the complexity of user-generated training conversations varies greatly, which leads to different difficulties for the agent to learn. Therefore, it is necessary for modeling dialogue complexity and make a reasonable learning schedule for efficiently training the agent. Towards that, we propose Scheduled Dialog Policy Learning, an auto- matic curriculum learning framework for joint- ing curriculum learning and policy optimiza- tion in the task-oriented dialog system. To our best knowledge, it is the first RL frame- work that improves dialogue policy learning by scheduling its learning process. Specifi- cally, we introduce an automatic measurement to evaluate the dialogue complexity, and based on this automatic measurement, we train the dialog agent from easy dialogues to complex ones. Experiments demonstrate that our ap- proach can be applied to the task-oriented dia- logue policy learning and outperforms the pre- vious state-of-the-art model, which increases 9.6% and 10.0% in the accuracy on the dialog success rate, respectively on the MultiWoz and Movie-Ticket Booking datasets. 1 Introduction Dialog policy learning is an important component of the task-oriented dialogue system, and it deter- mines the agent dialog action responding to the user. This learning process is often formulated as a reinforcement learning problem (Young et al., 2013; Levin et al., 1997; Dhingra et al., 2017; Li et al., 2017; Liu and Lane, 2017; Peng et al., 2018b; Su et al., 2018; Gao et al., 2019; Takanobu et al., ∗This work was done when Sihong Liu was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Corresponding author Figure 1: The comparison of the easy conversation (left) and the complex conversation (right) which are task-oriented dialogs between the user (U) and the sys- tem (S) sampling from MultiWoz. Comparing to the left conversation, the right one has more turns, intents, slots, and also switches between two domains: train (marked as blue) and restaurant (marked as dark). The right instance is apparently more complex. 2020), where users act as the environment and the agent learns the policy by interacting with users. Thus, the learning performance of the dialogue pol- icy depends much on users’ behaviors. However, due to the subjectivity and open-ended nature of human conversations, the complexity of training dialogues with different users varies greatly (Lison and Bibauw, 2017). Figure 1 shows dialogues with different complexities from Mul- tiWoz (Budzianowski et al., 2018) dataset. Com- paring to the left instance, the conversation in the right column has more turns, intents, slots, and …

Content truncated to 500 words