On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2431–2441, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems Pei-Hao Su, Milica Gaˇsi´c, Nikola Mrkˇsi´c, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen and Steve Young Department of Engineering, University of Cambridge, Cambridge, UK {phs26, mg436, nm480, lmr46, su259, djv27, thw28, sjy}@cam.ac.uk Abstract The ability to compute an accurate re- ward function is essential for optimising a dialogue policy via reinforcement learn- ing. In real-world applications, using ex- plicit user feedback as the reward sig- nal is often unreliable and costly to col- lect. This problem can be mitigated if the user’s intent is known in advance or data is available to pre-train a task suc- cess predictor off-line. In practice neither of these apply for most real world applica- tions. Here we propose an on-line learn- ing framework whereby the dialogue pol- icy is jointly trained alongside the reward model via active learning with a Gaussian process model. This Gaussian process op- erates on a continuous space dialogue rep- resentation generated in an unsupervised fashion using a recurrent neural network encoder-decoder. The experimental results demonstrate that the proposed framework is able to significantly reduce data annota- tion costs and mitigate noisy user feedback in dialogue policy learning. 1 Introduction Spoken Dialogue Systems (SDS) allow human- computer interaction using natural speech. They can be broadly divided into two categories: chat- oriented systems which aim to converse with users and provide reasonable contextually relevant re- sponses (Vinyals and Le, 2015; Serban et al., 2015), and task-oriented systems designed to as- sist users to achieve specific goals (e.g. find ho- tels, movies or bus schedules) (Daubigney et al., 2014; Young et al., 2013). The latter are typi- cally designed according to a structured ontology (or a database schema), which defines the domain Figure 1: An example of a task-oriented dialogue with a pre-defined task and the evaluation results. that the system can talk about. Teaching a system how to respond appropriately in a task-oriented SDS is non-trivial. This dialogue management task is often formulated as a manually defined di- alogue flow that directly determines the quality of interaction. More recently, dialogue management has been formulated as a reinforcement learning (RL) problem which can be automatically opti- mised (Levin and Pieraccini, 1997; Roy et al., 2000; Williams and Young, 2007; Young et al., 2013). In this framework, the system learns by a trial and error process governed by a poten- tially delayed learning objective defined by a re- ward function. A typical approach to defining the reward func- tion in a task-oriented dialogue system is to ap- ply a small per-turn penalty to encourage short dialogues and to give a large positive reward at the end of each successful interaction. Figure 1 is an example of a dialogue task which is typi- cally set for users who are being paid to converse with the system. When users …

Content truncated to 500 words