Learning Effective Multimodal Dialogue Strategies from Wizard-of-Oz Data: Bootstrapping and Evaluation

Proceedings of ACL-08: HLT, pages 638–646, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Learning Effective Multimodal Dialogue Strategies from Wizard-of-Oz data: Bootstrapping and Evaluation Verena Rieser School of Informatics University of Edinburgh Edinburgh, EH8 9LW, GB vrieser@inf.ed.ac.uk Oliver Lemon School of Informatics University of Edinburgh Edinburgh, EH8 9LW, GB olemon@inf.ed.ac.uk Abstract We address two problems in the field of au- tomatic optimization of dialogue strategies: learning effective dialogue strategies when no initial data or system exists, and evaluating the result with real users. We use Reinforcement Learning (RL) to learn multimodal dialogue strategies by interaction with a simulated envi- ronment which is “bootstrapped” from small amounts of Wizard-of-Oz (WOZ) data. This use of WOZ data allows development of op- timal strategies for domains where no work- ing prototype is available. We compare the RL-based strategy against a supervised strat- egy which mimics the wizards’ policies. This comparison allows us to measure relative im- provement over the training data. Our results show that RL significantly outperforms Super- vised Learning when interacting in simulation as well as for interactions with real users. The RL-based policy gains on average 50-times more reward when tested in simulation, and almost 18-times more reward when interacting with real users. Users also subjectively rate the RL-based policy on average 10% higher. 1 Introduction Designing a spoken dialogue system is a time- consuming and challenging task. A developer may spend a lot of time and effort anticipating the po- tential needs of a specific application environment and then deciding on the most appropriate system action (e.g. confirm, present items,. . . ). One of the key advantages of statistical optimisation methods, such as Reinforcement Learning (RL), for dialogue strategy design is that the problem can be formu- lated as a principled mathematical model which can be automatically trained on real data (Lemon and Pietquin, 2007; Frampton and Lemon, to appear). In cases where a system is designed from scratch, how- ever, there is often no suitable in-domain data. Col- lecting dialogue data without a working prototype is problematic, leaving the developer with a classic chicken-and-egg problem. We propose to learn dialogue strategies by simulation-based RL (Sutton and Barto, 1998), where the simulated environment is learned from small amounts of Wizard-of-Oz (WOZ) data. Us- ing WOZ data rather than data from real Human- Computer Interaction (HCI) allows us to learn op- timal strategies for domains where no working di- alogue system already exists. To date, automatic strategy learning has been applied to dialogue sys- tems which have already been deployed using hand- crafted strategies. In such work, strategy learning was performed based on already present extensive online operation experience, e.g. (Singh et al., 2002; Henderson et al., 2005). In contrast to this preced- ing work, our approach enables strategy learning in domains where no prior system is available. Opti- mised learned strategies are then available from the first moment of online-operation, and tedious hand- crafting of dialogue strategies is omitted. This inde- pendence from large amounts …

Content truncated to 500 words