Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems

Proceedings of NAACL-HLT 2018, pages 2060–2069 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems Bing Liu1∗, Gokhan T¨ur2, Dilek Hakkani-T¨ur2, Pararth Shah2, Larry Heck3† 1Carnegie Mellon University, Pittsburgh, PA, USA 2Google Research, Mountain View, CA,USA 3Samsung Research, Mountain View, CA, USA liubing@cmu.edu, {dilekh,pararth}@google.com, {gokhan.tur,larry.heck}@ieee.org Abstract In this work, we present a hybrid learn- ing method for training task-oriented dialogue systems through online user interactions. Pop- ular methods for learning task-oriented dia- logues include applying reinforcement learn- ing with user feedback on supervised pre- training models. Efficiency of such learning method may suffer from the mismatch of di- alogue state distribution between offline train- ing and online interactive learning stages. To address this challenge, we propose a hybrid imitation and reinforcement learning method, with which a dialogue agent can effectively learn from its interaction with users by learn- ing from human teaching and feedback. We design a neural network based task-oriented dialogue agent that can be optimized end-to- end with the proposed learning method. Ex- perimental results show that our end-to-end dialogue agent can learn effectively from the mistake it makes via imitation learning from user teaching. Applying reinforcement learn- ing with user feedback after the imitation learning stage further improves the agent’s ca- pability in successfully completing a task. 1 Introduction Task-oriented dialogue systems assist users to complete tasks in specific domains by understand- ing user’s request and aggregate useful informa- tion from external resources within several dia- logue turns. Conventional task-oriented dialogue systems have a complex pipeline (Rudnicky et al., 1999; Raux et al., 2005; Young et al., 2013) con- sisting of independently developed and modularly connected components for natural language un- derstanding (NLU) (Mesnil et al., 2015; Liu and Lane, 2016; Hakkani-T¨ur et al., 2016), dialogue state tracking (DST) (Henderson et al., 2014c; ∗ Work done while the author was an intern at Google. † Work done while at Google Research. Mrkˇsi´c et al., 2016), and dialogue policy learn- ing (Gasic and Young, 2014; Shah et al., 2016; Su et al., 2016, 2017). These system components are usually trained independently, and their optimiza- tion targets may not fully align with the overall system evaluation criteria (e.g. task success rate and user satisfaction). Moreover, errors made in the upper stream modules of the pipeline propa- gate to downstream components and get amplified, making it hard to track the source of errors. To address these limitations with the con- ventional task-oriented dialogue systems, re- cent efforts have been made in designing end- to-end learning solutions with neural network based methods. Both supervised learning (SL) based (Wen et al., 2017; Bordes and Weston, 2017; Liu and Lane, 2017a) and deep reinforce- ment learning (RL) based systems (Zhao and Es- kenazi, 2016; Li et al., 2017; Peng et al., 2017) have been studied in the literature. Comparing to chit-chat dialogue models that are usually trained offline using single-turn context-response pairs, task-oriented dialogue model involves …

Content truncated to 500 words