Using Reinforcement Learning to Build a Better Model of Dialogue State

Using Reinforcement Learning to Build a Better Model of Dialogue State Joel R. Tetreault University of Pittsburgh Learning Research and Development Center Pittsburgh PA, 15260, USA tetreaul@pitt.edu Diane J. Litman University of Pittsburgh Department of Computer Science & Learning Research and Development Center Pittsburgh PA, 15260, USA litman@cs.pitt.edu Abstract Given the growing complexity of tasks that spoken dialogue systems are trying to handle, Reinforcement Learning (RL) has been increasingly used as a way of au- tomatically learning the best policy for a system to make. While most work has focused on generating better policies for a dialogue manager, very little work has been done in using RL to construct a better dialogue state. This paper presents a RL approach for determining what dialogue features are important to a spoken dia- logue tutoring system. Our experiments show that incorporating dialogue factors such as dialogue acts, emotion, repeated concepts and performance play a signifi- cant role in tutoring and should be taken into account when designing dialogue sys- tems. 1 Introduction This paper presents initial research toward the long-term goal of designing a tutoring system that can effectively adapt to the student. While most work in Markov Decision Processes (MDPs) and spoken dialogue have focused on building better policies (Walker, 2000; Henderson et al., 2005), to date very little empirical work has tested the utility of adding specialized features to construct a better dialogue state. We wish to show that adding more complex factors to a representation of student state is a worthwhile pursuit, since it alters what action the tutor should make. The five dialogue factors we explore are dialogue acts, certainty level, frus- tration level, concept repetition, and student per- formance. All five are factors that are not just unique to the tutoring domain but are important to dialogue systems in general. Our results show that using these features, combined with the com- mon baseline of student correctness, leads to a sig- nificant change in the policies produced, and thus should be taken into account when designing a system. 2 Background We follow past lines of research (such as (Singh et al., 1999)) for describing a dialogue � as a tra- jectory within a Markov Decision Process (Sutton and Barto, 1998). A MDP has four main com- ponents: states, actions, a policy, which specifies what is the best action to take in a state, and a re- ward function which specifies the utility of each state and the process as a whole. Dialogue man- agement is easily described using a MDP because one can consider the actions as actions made by the system, the state as the dialogue context, and a reward which for many dialogue systems tends to be task completion success or dialogue length. Typically the state is viewed as a vector of features such as dialogue history, speech recognition con- fidence, etc. The goal of using MDPs is to determine the best policy � for a certain state and action space. That is, we wish to find …

Content truncated to 500 words