Affordable On-line Dialogue Policy Learning

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2200–2209 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Affordable On-line Dialogue Policy Learning Cheng Chang∗, Runzhe Yang∗, Lu Chen, Xiang Zhou and Kai Yu Key Lab. of Shanghai Education Commission for Intelligent Interaction and Cognitive Eng. SpeechLab, Department of Computer Science and Engineering Brain Science and Technology Research Center Shanghai Jiao Tong University, Shanghai, China {cheng.chang,yang runzhe,chenlusz,owenzx,kai.yu}@sjtu.edu.cn Abstract The key to building an evolvable dialogue system in real-world scenarios is to en- sure an affordable on-line dialogue policy learning, which requires the on-line learn- ing process to be safe, efficient and eco- nomical. But in reality, due to the scarcity of real interaction data, the dialogue sys- tem usually grows slowly. Besides, the poor initial dialogue policy easily leads to bad user experience and incurs a failure of attracting users to contribute training data, so that the learning process is un- sustainable. To accurately depict this, t- wo quantitative metrics are proposed to as- sess safety and efficiency issues. For solv- ing the unsustainable learning problem, we proposed a complete companion teach- ing framework incorporating the guidance from the human teacher. Since the human teaching is expensive, we compared vari- ous teaching schemes answering the ques- tion how and when to teach, to economi- cally utilize teaching budget, so that make the online learning process affordable. 1 Introduction A task-oriented dialogue system is designed for interacting with humans users to accomplish sev- eral predefined domains or tasks (Young et al., 2013; Daubigney et al., 2012). Dialogue Man- ager is the core component in a typical dialogue system, which controls the flow of dialogue by a state tracker and a policy module (Levin et al., 1997). The state tracker tracks the internal s- tate of the system while the policy module de- cides the response to the user according to the s- tatus of states (Sun et al., 2014a; Thomson and ∗ Both authors contributed equally to this work. Young, 2010). The approaches of constructing a policy module can be divided into two cate- gories: rule-based and statistical. Rule-based poli- cies are usually hand-crafted by domain experts which means they are inconvenient and difficult to be optimized (Williams and Young, 2007; Wang and Lemon, 2013). In recent mainstream statisti- cal studies, Partially Observable Markov Decision Process (POMDP) framework has been applied to model dialogue management with unobservable s- tates, where policy training can be formulated as a Reinforcement Learning (RL) problem, which enables the policy to be optimized automatically (Kaelbling et al., 1998; Arnold, 1998; Young et al., 2013). Though RL-based approaches have the poten- tial to improve themselves as they interact more with human users and achieve better performance than rule-based approaches, they are rarely used in real-world applications, especially in on-line sce- narios, since the training process is unsustainable. Vicious Cycle: Unsustainable On-line Learning Poor (initial) Policy Bad User Experience Insufficient Real User (Data) Unsafe Policy Behavior (Solvable) ✔ Individual Rationality (Unsolvable) ✘ …

Content truncated to 500 words