Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3537–3546 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3537 Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems Ziming Li 1, Julia Kiseleva 2, Maarten de Rijke 1 1University of Amsterdam, 2Microsoft Research {z.li,m.derijke@@uva.nl}, julia.kiseleva@microsoft.com Abstract Dialogue policy learning for Task-oriented Dialogue Systems (TDSs) has enjoyed great progress recently mostly through employing Reinforcement Learning (RL) methods. How- ever, these approaches have become very so- phisticated. It is time to re-evaluate it. Are we really making progress developing dia- logue agents only based on RL? We demon- strate how (1) traditional supervised learning together with (2) a simulator-free adversarial learning method can be used to achieve perfor- mance comparable to state-of-the-art (SOTA) RL-based methods. First, we introduce a sim- ple dialogue action decoder to predict the ap- propriate actions. Then, the traditional multi- label classification solution for dialogue pol- icy learning is extended by adding dense lay- ers to improve the dialogue agent performance. Finally, we employ the Gumbel-Softmax esti- mator to alternatively train the dialogue agent and the dialogue reward model without using RL. Based on our extensive experimentation, we can conclude the proposed methods can achieve more stable and higher performance with fewer efforts, such as the domain knowl- edge required to design a user simulator and the intractable parameter tuning in reinforce- ment learning. Our main goal is not to beat RL with supervised learning, but to demonstrate the value of rethinking the role of RL and su- pervised learning in optimizing TDSs. 1 Introduction The aim of dialogue policies in Task-oriented Dia- logue System (TDS) is to select appropriate actions at each time step according to the current context of the conversation and user feedback (Chen et al., 2017). In early work, dialogue policies were manu- ally designed as a set of rules that map the dialogue context to a corresponding system action (Weizen- baum, 1966). The ability of rule-based solutions is limited by the domain complexity and task scalabil- ity. Moreover, the design and maintenance of these rules require a lot of effort and domain knowledge. Due to recent advantages in deep learning and the availability of labeled conversational datasets, supervised learning can be employed for dialogue policy training to overcome the disadvantages of rule-based systems. The downside of the super- vised learning approach is that the dialogues ob- served in the datasets are unlikely to represent all possible conversation scenarios; in some extreme cases, the required conversational dataset cannot be collected or acquiring it might cost-prohibitive. The success of RL in other areas holds promises for dialogue Policy Learning (PL) (Williams and Young, 2007). Using RL techniques, we can train dialogue policies and optimize automatically, from scratch and utilizing interactions with users (Gaˇsi´c and Young, 2014; Su et al., 2017). In RL-based solutions, the dialogue system takes actions that are controlled by the dialogue policy, and user feed- back (the reward signal), which is provided when the dialogue is finished, is …

Content truncated to 500 words