Clipping Loops for Sample-Efficient Dialogue Policy Optimisation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3420–3428 June 6–11, 2021. ©2021 Association for Computational Linguistics 3420 Clipping Loops for Sample-Efficient Dialogue Policy Optimisation Yen-Chen Wu University of Cambridge ycw30@cam.ac.uk Carl Edward Rasmussen University of Cambridge cer54@cam.ac.uk Abstract Training dialogue agents requires a large num- ber of interactions with users: agents have no idea about which responses are bad among a lengthy dialogue. In this paper, we propose loop-clipping policy optimisation (LCPO) to eliminate useless responses. LCPO consists of two stages: loop clipping and advantage clip- ping. In loop clipping, we clip off useless re- sponses (called loops) from dialogue history (called trajectories). The clipped trajectories are more succinct than the original ones, and the estimation of state-value is more accurate. Second, in advantage clipping, we estimate and clip the advantages of useless responses and normal ones separately. The clipped ad- vantage distinguishes useless actions from oth- ers and reduces the probabilities of useless ac- tions efficiently. In experiments on Cambridge Restaurant Dialogue System, LCPO uses only 260 training dialogues to achieve 80% success rate, while PPO baseline requires 2160 dia- logues. Besides, LCPO receives 3.7/5 scores in human evaluation where the agent interac- tively collects 100 real-user dialogues in the training phase. 1 Introduction Based on dialogue policies, task-oriented dialogue systems decide when and how to give or request in- formation from users. Learning dialogue policies is often formulated as a reinforcement learning (RL) problem since we usually receive feedback from users for the whole dialogue but not the correct answer for a single response (Young et al., 2013; Levin et al., 1997). With high-capacity of function approximation, deep reinforcement learning has been widely applied to dialogue policy optimisa- tion (Su et al., 2016; Li et al., 2016; Casanueva et al., 2017). Typically, when applying deep re- inforcement learning for dialogue policy manage- ment, more than thousands of dialogues are re- quired to reach convergence (Casanueva et al., 2017). However, requiring thousands of human dialogues during training is quite impractical for most academic or real-life scenarios. Users might lose patience and exhibit different behaviour dur- ing training. Therefore, in most prior work, the agents are trained via simulated users instead of real ones (Liu et al., 2018; Gao et al., 2018). Model-based reinforcement learning (MBRL) is commonly applied to make dialogue policy opti- misation sample-efficient. MBRL approaches for dialogue management build a user model to predict users’ behaviour (Wu et al., 2020b,a; Peng et al., 2018; Su et al., 2018; Wu et al., 2019; Zhang et al., 2019). Using the user model, DDQ (Peng et al., 2018) generates pseudo-data. The accuracy of the user model strongly affects the quality of generated pseudo-data. If the behaviour of pseudo-data is far from real users’ behaviour, dialogue policies learnt from these data might not be optimal (Su et al., 2018). Manipulating when to use how much data in experience buffers becomes critical in these approaches. Trainable-action-mask (TAM) (Wu et …

Content truncated to 500 words