Budgeted Policy Learning for Task-Oriented Dialogue Systems

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3742–3751 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3742 Budgeted Policy Learning for Task-Oriented Dialogue Systems Zhirui Zhang† Xiujun Li‡§ Jianfeng Gao‡ Enhong Chen† †University of Science and Technology of China ‡Microsoft Research AI §University of Washington †zrustc11@gmail.com †cheneh@ustc.edu.cn ‡{xiul,jfgao}@microsoft.com Abstract This paper presents a new approach that ex- tends Deep Dyna-Q (DDQ) by incorporating a Budget-Conscious Scheduling (BCS) to best utilize a fixed, small amount of user interac- tions (budget) for learning task-oriented dia- logue agents. BCS consists of (1) a Poisson- based global scheduler to allocate budget over different stages of training; (2) a controller to decide at each training step whether the agent is trained using real or simulated experiences; (3) a user goal sampling module to generate the experiences that are most effective for pol- icy learning. Experiments on a movie-ticket booking task with simulated and real users show that our approach leads to significant im- provements in success rate over the state-of- the-art baselines given the fixed budget. 1 Introduction There has been a growing interest in exploiting reinforcement learning (RL) for dialogue policy learning in task-oriented dialogue systems (Levin et al., 1997; Williams, 2008; Young et al., 2013; Fatemi et al., 2016; Zhao and Esk´enazi, 2016; Su et al., 2016; Li et al., 2017; Williams et al., 2017; Dhingra et al., 2017; Budzianowski et al., 2017; Chang et al., 2017; Liu and Lane, 2017; Liu et al., 2018; Gao et al., 2019). This is a challenging machine learning task because an RL learner re- quires real users to interact with a dialogue agent constantly to provide feedback. The process in- curs significant real-world cost for complex tasks, such as movie-ticket booking and travel planning, which require exploration in a large state-action space. In reality, we often need to develop a dialogue agent with some fixed, limited budget due to lim- ited project funding, conversational data, and de- velopment time. Specifically, in this study we measure budget in terms of the number of real user interactions. That is, we strive to optimize a dia- logue agent via a fixed, small number of interac- tions with real users. One common strategy is to leverage a user sim- ulator built on human conversational data (Schatz- mann et al., 2007; Li et al., 2016). However, due to design bias and the limited amounts of pub- licly available human conversational data for train- ing the simulator, there always exists discrepan- cies between the behaviors of real and simulated users, which inevitably leads to a sub-optimal dia- logue policy. Another strategy is to integrate plan- ning into dialogue policy learning, as the Deep Dyna-Q (DDQ) framework (Peng et al., 2018), which effectively leverages a small number of real experiences to learn a dialogue policy efficiently. In DDQ, the limited amounts of real user experi- ences are utilized for: (1) training a world model to mimic real user behaviors and generate simulated …

Content truncated to 500 words