Learning Efficient Dialogue Policy from Demonstrations through Shaping

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6355–6365 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6355 Learning Efficient Dialogue Policy from Demonstrations through Shaping Huimin Wang♣† Baolin Peng♠†∗ Kam-Fai Wong♣ ♠Microsoft Research ♣The Chinese University of Hong Kong bapeng@microsoft.com {hmwang,kfwong}@se.cuhk.edu.hk Abstract Training a task-oriented dialogue agent with reinforcement learning is prohibitively expen- sive since it requires a large volume of inter- actions with users. Human demonstrations can be used to accelerate learning progress. However, how to effectively leverage demon- strations to learn dialogue policy remains less explored. In this paper, we present that ef- ficiently learns dialogue policy from demon- strations through policy shaping and reward shaping. We use an imitation model to dis- till knowledge from demonstrations, based on which policy shaping estimates feedback on how the agent should act in policy space. Re- ward shaping is then incorporated to bonus state-actions similar to demonstrations explic- itly in value space encouraging better explo- ration. The effectiveness of the proposed S2Agent is demonstrated in three dialogue do- mains and a challenging domain adaptation task with both user simulator evaluation and human evaluation. 1 Introduction With the flourishment of conversational assistants in daily life (like Google Assistant, Amazon Alexa, Apple Siri, and Microsoft Cortana), task-oriented dialogues that are able to serve users on certain tasks have increasingly attracted research efforts. Dialogue policy optimization is one of the most critical tasks of dialogue modeling. One of the most straightforward approaches is the rule-based method, which contains a set of expert-defined rules for dialogue modeling. Though rule-based dialogue systems have a reasonable performance in some scenarios, handcrafting such kinds of rules is time-consuming and not scalable. Recently, dialogue policy learning is formulated as a reinforcement learning (RL) problem and tack- led with deep RL models (Li et al., 2017; Lipton ∗Corresponding author †Equal Contribution et al., 2018; Peng et al., 2017). It has shown great potentials of using the RL-based method for build- ing robust dialogue systems automatically. How- ever, due to its interactive nature, RL-based agents demand of an environment to operate in. As illus- trated in Figure 1, RL-based dialogue agents need to interact with human users and update its pol- icy in an online fashion requiring that the agents have a good online performance from the start of training. In addition, one of the biggest challenges of RL approaches is reward sparsity issue, which leads to exploration in large action space inefficient. As a consequence, training RL-based agents ex- pects a prohibitively large number of interactions to achieve acceptable performance, which may incur a significant amount of expense (Pietquin et al., 2011; Lipton et al., 2016; Peng et al., 2018b). Several at- tempts are made to improve learning efficiency and tackle reward sparsity issues. Different types of heuristics has been proposed in the form of intrin- sic rewards to guide exploration more efficiently (Lipton et al., 2016; Mohamed and Rezende, 2015; Peng et al., 2017, 2018a; Takanobu et al., 2019). When …

Content truncated to 500 words