Combining Reinformation Learning with Information-State Update Rules

Proceedings of NAACL HLT 2007, pages 268–275, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Combining Reinforcement Learning with Information-State Update Rules∗ Peter A. Heeman Center for Spoken Language Understanding Oregon Health & Science University Beaverton OR, 97006, USA heeman@cslu.ogi.edu Abstract Reinforcement learning gives a way to learn under what circumstances to per- form which actions. However, this ap- proach lacks a formal framework for spec- ifying hand-crafted restrictions, for speci- fying the effects of the system actions, or for specifying the user simulation. The in- formation state approach, in contrast, al- lows system and user behavior to be spec- ified as update rules, with preconditions and effects. This approach can be used to specify complex dialogue behavior in a systematic way. We propose combining these two approaches, thus allowing a for- mal specification of the dialogue behavior, and allowing hand-crafted preconditions, with remaining ones determined via rein- forcement learning so as to minimize dia- logue cost. 1 Introduction Two different approaches have become popular for building spoken dialogue systems. The first is the symbolic reasoning approach. Speech actions are defined in a formal logic, in terms of the situations in which they can be applied, and what effect they will have on the speaker’s and the listener’s mental state (Cohen and Perrault, 1979; Allen and Perrault, 1980). One of these approaches is the information state (IS) approach (Larsson and Traum, 2000). The knowledge of the agent is formalized as the state. The IS state is updated by way of update rules, which have preconditions and effects. The precondi- tions specify what must be true of the state in order ∗The author wishes to thank Fan Yang and Michael En- glish for helpful conversations. Funding from the National Science Foundation under grant IIS-0326496 is gratefully ac- knowledged. to apply the rule. The effects specify how the state changes as a result of applying the rule. At a mini- mum, two sets of update rules are used: one set, un- derstanding rules, specify the effect of an utterance on the agent’s state and a second set, action rules, specify which speech action can be performed next. For example, a precondition for asking a question is that the agent does not know the answer to the ques- tion. An effect of an answer to a question is that the hearer now knows the answer. One problem with this approach is that although necessary precondi- tions for speech actions are easy to code, there are typically many speech actions that can be applied at any point in a dialogue. Determining which one is the optimal one is a daunting task for the dialogue designer. The second approach for building spoken dia- logue systems is to use reinforcement learning (RL) to automatically determine what action to perform in each different dialogue state so as to minimize some cost function (e.g. Walker, 2000; Levin et al., 2000). The problem with this approach, however, is that it lacks the framework of IS to specify the …

Content truncated to 500 words