Deep Active Learning for Dialogue Generation

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 78–83, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics Deep Active Learning for Dialogue Generation Nabiha Asghar†, Pascal Poupart†, Xin Jiang‡, Hang Li‡ † Cheriton School of Computer Science, University of Waterloo, Canada {nasghar,ppoupart}@uwaterloo.ca ‡Noah’s Ark Lab, Huawei Technologies, Hong Kong {jiang.xin,hangli.hl}@huawei.com Abstract We propose an online, end-to-end, neural generative conversational model for open- domain dialogue. It is trained using a unique combination of offline two-phase supervised learning and online human-in- the-loop active learning. While most ex- isting research proposes offline supervi- sion or hand-crafted reward functions for online reinforcement, we devise a novel interactive learning mechanism based on hamming-diverse beam search for re- sponse generation and one-character user- feedback at each step. Experiments show that our model inherently promotes the generation of semantically relevant and interesting responses, and can be used to train agents with customized personas, moods and conversational styles. 1 Introduction Several recent works propose neural generative conversational agents (CAs) for open-domain and task-oriented dialogue (Shang et al., 2015; Sor- doni et al., 2015; Vinyals and Le, 2015; Serban et al., 2016, 2017; Wen et al., 2016; Shen et al., 2017; Eric and Manning, 2017a,b). These mod- els typically use LSTM encoder-decoder architec- tures (e.g. the sequence-to-sequence (Seq2Seq) framework (Sutskever et al., 2014)), which are lin- guistically robust but can often generate short, dull and inconsistent responses (Serban et al., 2016; Li et al., 2016a). Researchers are now exploring Deep Reinforcement Learning (DRL) to address the hard problems of NLU and NLG in dialogue generation. In most of the existing works, the re- ward function is hand-crafted, and is either spe- cific to the task to be completed, or is based on a few desirable developer-defined conversational properties. In this work we demonstrate how online Deep Active Learning can be integrated with standard neural network based dialogue systems to enhance their open-domain conversational skills. The ar- chitectural backbone of our model is the Seq2Seq framework, which initially undergoes offline su- pervised learning on two different types of con- versational datasets. We then initiate an on- line active learning phase to interact with human users for incremental model improvement, where a unique single-character1 user-feedback mecha- nism is used as a form of reinforcement at each turn in the dialogue. The intuition is to rely on this all-encompassing human-centric ‘reinforcement’ mechanism, instead of defining hand-crafted re- ward functions that individually try to capture each of the many subtle conversational properties. This mechanism inherently promotes interesting and relevant responses by relying on the humans’ far superior conversational prowess. 2 Related Work & Contributions DRL-based dialogue generation is a relatively new research paradigm that is most relevant to our work. For task-specific dialogue (Su et al., 2016; Zhao and Eskenazi, 2016; Cuay´ahuitl et al., 2016; Williams and Zweig, 2016; Li et al., 2017b,c; Peng et al., 2017), the reward function is usually based on task completion rate, and thus is easy to define. For the …

Content truncated to 500 words