Learning to Compose Effective Strategies from a Library of Dialogue Components

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 792–799, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Learning to Compose Effective Strategies from a Library of Dialogue Components Martijn Spitters† Marco De Boni‡ Jakub Zavrel† Remko Bonnema† † Textkernel BV, Nieuwendammerkade 28/a17, 1022 AB Amsterdam, NL {spitters,zavrel,bonnema}@textkernel.nl ‡ Unilever Corporate Research, Colworth House, Sharnbrook, Bedford, UK MK44 1LQ marco.de-boni@unilever.com Abstract This paper describes a method for automat- ically learning effective dialogue strategies, generated from a library of dialogue content, using reinforcement learning from user feed- back. This library includes greetings, so- cial dialogue, chit-chat, jokes and relation- ship building, as well as the more usual clar- ification and verification components of dia- logue. We tested the method through a mo- tivational dialogue system that encourages take-up of exercise and show that it can be used to construct good dialogue strategies with little effort. 1 Introduction Interactions between humans and machines have be- come quite common in our daily life. Many ser- vices that used to be performed by humans have been automated by natural language dialogue sys- tems, including information seeking functions, as in timetable or banking applications, but also more complex areas such as tutoring, health coaching and sales where communication is much richer, embed- ding the provision and gathering of information in e.g. social dialogue. In the latter category of dia- logue systems, a high level of naturalness of interac- tion and the occurrence of longer periods of satisfac- tory engagement with the system are a prerequisite for task completion and user satisfaction. Typically, such systems are based on a dialogue strategy that is manually designed by an expert based on knowledge of the system and the domain, and on continuous experimentation with test users. In this process, the expert has to make many de- sign choices which influence task completion and user satisfaction in a manner which is hard to assess, because the effectiveness of a strategy depends on many different factors, such as classification/ASR performance, the dialogue domain and task, and, perhaps most importantly, personality characteris- tics and knowledge of the user. We believe that the key to maximum dialogue ef- fectiveness is to listen to the user. This paper de- scribes the development of an adaptive dialogue sys- tem that uses the feedback of users to automatically improve its strategy. The system starts with a library of generic and task-/domain-specific dialogue com- ponents, including social dialogue, chit-chat, enter- taining parts, profiling questions, and informative and diagnostic parts. Given this variety of possi- ble dialogue actions, the system can follow many different strategies within the dialogue state space. We conducted training sessions in which users inter- acted with a version of the system which randomly generates a possible dialogue strategy for each in- teraction (restricted by global dialogue constraints). After each interaction, the users were asked to re- ward different aspects of the conversation. We ap- plied reinforcement learning to use this feedback to compute the optimal dialogue policy. …

Content truncated to 500 words