Guided Dialogue Policy Learning without Adversarial Learning in the Loop

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2308–2317 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2308 Guided Dialogue Policy Learning without Adversarial Learning in the Loop Ziming Li 1, Sungjin Lee 2, Baolin Peng 3, Jinchao Li 3, Julia Kiseleva 3, Maarten de Rijke 1, Shahin Shayandeh 3, Jianfeng Gao 3 1University of Amsterdam, 2Amazon, 3Microsoft Research z.li@@uva.nl, sungjinl@amazon.com, baolin.peng@microsoft.com Abstract Reinforcement Learning (RL) methods have emerged as a popular choice for training an efficient and effective dialogue policy. How- ever, these methods suffer from sparse and un- stable reward signals returned by a user simu- lator only when a dialogue finishes. Besides, the reward signal is manually designed by hu- man experts, which requires domain knowl- edge. Recently, a number of adversarial learn- ing methods have been proposed to learn the reward function together with the dialogue policy. However, to alternatively update the dialogue policy and the reward model on the fly, we are limited to policy-gradient-based algorithms, such as REINFORCE and PPO. Moreover, the alternating training of a dia- logue agent and the reward model can easily get stuck in local optima or result in mode collapse. To overcome the listed issues, we propose to decompose the adversarial train- ing into two steps. First, we train the dis- criminator with an auxiliary dialogue gener- ator and then incorporate a derived reward model into a common RL method to guide the dialogue policy learning. This approach is ap- plicable to both on-policy and off-policy RL methods. Based on our extensive experimen- tation, we can conclude the proposed method: (1) achieves a remarkable task success rate us- ing both on-policy and off-policy RL meth- ods; and (2) has potential to transfer knowl- edge from existing domains to a new domain. 1 Introduction Task-oriented Dialogue Systems (TDSs), such as Siri, Google Assistant, and Amazon Alexa, aim to offer users assistance with completing tasks. TDSs need dialogue policies to select appropri- ate actions at each dialogue step according to the current context of the conversation (Chen et al., 2017). The development of RL in robotics and other domains has brought a new view on learn- ing dialogue policies (Williams and Young, 2007; Gaˇsi´c and Young, 2014; Su et al., 2017): it allows us to train with far more data than can be feasibly collected from actual users. The aim of TDSs is to maximize positive user feedback. TDSs based on RL are amenable to training with user simulators instead of real humans (Schatzmann et al., 2007; Li et al., 2016). User simulators rely on a reward function that scores system actions given dialogue context (Peng et al., 2018b; Williams et al., 2017; Dhingra et al., 2016; Su et al., 2016). The most straightforward way to design a dia- logue reward function is to score the agent based on the dialogue status in a rule-based fashion: if the dialogue ends successfully, a large positive re- ward will be returned; if the dialogue fails, the re- ward will …

Content truncated to 500 words