Comparing the Utility of State Features in Spoken Dialogue Using Reinforcement Learning

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 272–279, New York, June 2006. c⃝2006 Association for Computational Linguistics Comparing the Utility of State Features in Spoken Dialogue Using Reinforcement Learning Joel R. Tetreault University of Pittsburgh Learning Research and Development Center Pittsburgh PA, 15260, USA tetreaul@pitt.edu Diane J. Litman University of Pittsburgh Department of Computer Science Learning Research and Development Center Pittsburgh PA, 15260, USA litman@cs.pitt.edu Abstract Recent work in designing spoken dialogue systems has focused on using Reinforce- ment Learning to automatically learn the best action for a system to take at any point in the dialogue to maximize dia- logue success. While policy development is very important, choosing the best fea- tures to model the user state is equally im- portant since it impacts the actions a sys- tem should make. In this paper, we com- pare the relative utility of adding three fea- tures to a model of user state in the do- main of a spoken dialogue tutoring sys- tem. In addition, we also look at the ef- fects of these features on what type of a question a tutoring system should ask at any state and compare it with our previ- ous work on using feedback as the system action. 1 Introduction A host of issues confront spoken dialogue system designers, such as choosing the best system action to perform given any user state, and also selecting the right features to best represent the user state. While recent work has focused on using Reinforcement Learning (RL) to address the first issue (such as (Walker, 2000), (Henderson et al., 2005), (Williams et al., 2005a)), there has been very little empirical work on the issue of feature selection in prior RL ap- proaches to dialogue systems. In this paper, we use a corpus of dialogues of humans interacting with a spoken dialogue tutoring system to show the com- parative utility of adding the three features of con- cept repetition, frustration level, and student perfor- mance. These features are not just unique to the tu- toring domain but are important to dialogue systems in general. Our empirical results show that these fea- tures all lead to changes in what action the system should take, with concept repetition and frustration having the largest effects. This paper extends our previous work (Tetreault and Litman, 2006) which first presented a method- ology for exploring whether adding more complex features to a representation of student state will ben- eficially alter tutor actions with respect to feedback. Here we present an empirical method of comparing the effects of each feature while also generalizing our findings to a different action choice of what type of follow-up question should a tutor ask the student (as opposed to what type of feedback should the tu- tor give). In complex domains such as tutoring, test- ing different policies with real or simulated students can be time consuming and costly so it is important to properly choose the best features …

Content truncated to 500 words