Agent-Aware Dropout DQN for Safe and Efficient On-line Dialogue Policy Learning

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2454–2464 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Agent-Aware Dropout DQN for Safe and Efficient On-line Dialogue Policy Learning Lu Chen and Xiang Zhou and Cheng Chang and Runzhe Yang and Kai Yu Key Lab. of Shanghai Education Commission for Intelligent Interaction and Cognitive Eng. SpeechLab, Department of Computer Science and Engineering Brain Science and Technology Research Center Shanghai Jiao Tong University, Shanghai, China {chenlusz,owenzx,cheng.chang,yang runzhe,kai.yu}@sjtu.edu.cn Abstract Hand-crafted rules and reinforcement learning (RL) are two popular choices to obtain dialogue policy. The rule-based policy is often reliable within predefined scope but not self-adaptable, whereas RL is evolvable with data but often suffers from a bad initial performance. We em- ploy a companion learning framework to integrate the two approaches for on-line dialogue policy learning, in which a pre- defined rule-based policy acts as a teacher and guides a data-driven RL system by giving example actions as well as ad- ditional rewards. A novel agent-aware dropout Deep Q-Network (AAD-DQN) is proposed to address the problem of when to consult the teacher and how to learn from the teacher’s experiences. AAD- DQN, as a data-driven student policy, pro- vides (1) two separate experience memo- ries for student and teacher, (2) an uncer- tainty estimated by dropout to control the timing of consultation and learning. Sim- ulation experiments showed that the pro- posed approach can significantly improve both safety and efficiency of on-line pol- icy optimization compared to other com- panion learning approaches as well as su- pervised pre-training using static dialogue corpus. 1 Introduction A task-oriented spoken dialogue system (SDS) is a system that can continuously interact with a human to accomplish a predefined task through speech. Dialogue manager, which maintains the dialogue state and decides how to respond, is the core of an SDS. In this paper, we focus on the dia- logue policy. At the early research, the spoken dialogue sys- tems assume observable dialogue states. Dialogue policy is simply a set of hand-crafted mapping rules from state to machine action. This is referred to as rule-based policy, which often has acceptable performance but has no ability of self-adaption. Nowadays rule-based policy is popular in com- mercial dialogue systems. However, in real world scenarios, unpredictable user behavior, inevitable automatic speech recog- nition, and spoken language understanding errors make it difficult to maintain the true dialogue state and make the decision. Hence, in recent years, there is a research trend towards statistical dia- logue management. A well-founded theory for this is the partially observable Markov decision process (POMDP) (Kaelbling et al., 1998), which can provide robustness to errors from the input module and automatic policy optimization by re- inforcement learning. Most POMDP based policy learning research is usually carried out using ei- ther user simulator or employed users (Williams and Young, 2007; Young et al., 2010). The trained policy is not guaranteed to work well in real world scenarios. Therefore, on-line policy training has been …

Content truncated to 500 words