Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 665–677 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1062 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 665–677 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1062 Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning Jason D. Williams Microsoft Research jason.williams@microsoft.com Kavosh Asadi Brown University kavosh@brown.edu Geoffrey Zweig∗ Microsoft Research g2zweig@gmail.com Abstract End-to-end learning of recurrent neural networks (RNNs) is an attractive solu- tion for dialog systems; however, cur- rent techniques are data-intensive and re- quire thousands of dialogs to learn sim- ple behaviors. We introduce Hybrid Code Networks (HCNs), which combine an RNN with domain-specific knowledge encoded as software and system action templates. Compared to existing end-to- end approaches, HCNs considerably re- duce the amount of training data required, while retaining the key benefit of inferring a latent representation of dialog state. In addition, HCNs can be optimized with su- pervised learning, reinforcement learning, or a mixture of both. HCNs attain state- of-the-art performance on the bAbI dialog dataset (Bordes and Weston, 2016), and outperform two commercially deployed customer-facing dialog systems. 1 Introduction Task-oriented dialog systems help a user to ac- complish some goal using natural language, such as making a restaurant reservation, getting techni- cal support, or placing a phonecall. Historically, these dialog systems have been built as a pipeline, with modules for language understanding, state tracking, action selection, and language genera- tion. However, dependencies between modules in- troduce considerable complexity – for example, it is often unclear how to define the dialog state and what history to maintain, yet action selection re- lies exclusively on the state for input. Moreover, training each module requires specialized labels. ∗Currently at JPMorgan Chase Recently, end-to-end approaches have trained recurrent neural networks (RNNs) directly on text transcripts of dialogs. A key benefit is that the RNN infers a latent representation of state, obviat- ing the need for state labels. However, end-to-end methods lack a general mechanism for injecting domain knowledge and constraints. For example, simple operations like sorting a list of database re- sults or updating a dictionary of entities can ex- pressed in a few lines of software, yet may take thousands of dialogs to learn. Moreover, in some practical settings, programmed constraints are es- sential – for example, a banking dialog system would require that a user is logged in before they can retrieve account information. This paper presents a model for end-to-end learning, called Hybrid Code Networks (HCNs) which addresses these problems. In addition to learning an RNN, HCNs also allow a developer to express domain knowledge via software and ac- tion templates. Experiments show that, compared to existing recurrent end-to-end techniques, HCNs achieve the same performance with considerably less training data, while retaining the key benefit of end-to-end trainability. Moreover, the neural network can be trained with supervised learning or reinforcement learning, …

Content truncated to 500 words