Learning Optimal Dialogue Management Rules by Using Reinforcement Learning and Inductive Logic Programming

Learning optimal dialogue management rules by using reinforcement learning and inductive logic programming Renaud Lecœuche Microsoft Corporation Compass House, 80-82 Newmarket road Cambridge CB5 8DZ, United Kingdom renaudle@microsoft.com Abstract Developing dialogue systems is a complex pro- cess. In particular, designing efficient dialogue management strategies is often difficult as there are no precise guidelines to develop them and no sure test to validate them. Several suggestions have been made recently to use reinforcement learning to search for the optimal management strategy for specific dialogue situations. These approaches have produced interesting results, including applications involving real world dia- logue systems. However, reinforcement learning suffers from the fact that it is state based. In other words, the optimal strategy is expressed as a decision table specifying which action to take in each specific state. It is therefore diffi- cult to see whether there is any generality across states. This limits the analysis of the optimal strategy and its potential for re-use in other di- alogue situations. In this paper we tackle this problem by learning rules that generalize the state-based strategy. These rules are more read- able than the underlying strategy and therefore easier to explain and re-use. We also investi- gate the capability of these rules in directing the search for the optimal strategy by looking for generalization whilst the search proceeds. 1 Introduction As dialogue systems become ubiquitous, dia- logue management strategies are receiving more and more attention. They define the system be- havior and mainly determine how well or badly it is perceived by users. Generic methodolo- gies exist for developing and testing manage- ment strategies. Many of these take a user- centric approach based on Wizard of Oz stud- ies and iterative design (Bernsen et al., 1998). However there are still no precise guidelines about when to use specific techniques such as mixed-initiative. Reinforcement learning has been used in several recent approaches to search for the optimal dialogue management strategy for specific dialogue situations (Levin and Pier- accini, 1997; Litman et al., 2000; Singh et al., 2000; Walker, 2000). In these approaches, a dia- logue is seen as a walk through a series of states, from an initial state when the dialogue begins until a terminal state when the dialogue ends. The actions of the dialogue manager as well as those of the user influence the transitions be- tween states. Each transition is associated with a reward, which expresses how good or bad it was to make that transition. A dialogue strat- egy is then seen as a Markov Decision Process (Levin et al., 1998). Reinforcement learning can be used in this framework to search for an op- timal strategy, i.e., a strategy that makes the expected sum of rewards maximal for all the training dialogues. The main idea behind re- inforcement learning is to explore the space of possible dialogues and select the strategy which optimizes the expected rewards (Mitchell, 1997, ch. 13). Once the optimal strategy has been found, it can be implemented in the final sys- …

Content truncated to 500 words