Modelling User Satisfaction and Student Learning in a Spoken Dialogue Tutoring System with Generic, Tutoring, and User Affect Parameters

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 264–271, New York, June 2006. c⃝2006 Association for Computational Linguistics Modelling User Satisfaction and Student Learning in a Spoken Dialogue Tutoring System with Generic, Tutoring, and User Affect Parameters Kate Forbes-Riley Learning Research & Development Ctr University of Pittsburgh Pittsburgh, PA 15260 forbesk@cs.pitt.edu Diane J. Litman Learning Research & Development Ctr Dept. of Computer Science University of Pittsburgh Pittsburgh, PA 15260 litman@cs.pitt.edu Abstract We investigate using the PARADISE framework to develop predictive models of system performance in our spoken di- alogue tutoring system. We represent per- formance with two metrics: user satis- faction and student learning. We train and test predictive models of these met- rics in our tutoring system corpora. We predict user satisfaction with 2 parameter types: 1) system-generic, and 2) tutoring- specific. To predict student learning, we also use a third type: 3) user affect. Al- hough generic parameters are useful pre- dictors of user satisfaction in other PAR- ADISE applications, overall our parame- ters produce less useful user satisfaction models in our system. However, generic and tutoring-specific parameters do pro- duce useful models of student learning in our system. User affect parameters can in- crease the usefulness of these models. 1 Introduction In recent years the development of spoken dialogue tutoring systems has become more prevalent, in an attempt to close the performance gap between hu- man and computer tutors (Mostow and Aist, 2001; Pon-Barry et al., 2004; Litman et al., 2006). Student learning is a primary metric for evaluating the per- formance of these systems; it can be measured, e.g., by comparing student pretests taken prior to system use with posttests taken after system use. In other types of spoken dialogue systems, the user’s subjective judgments about using the system are often considered a primary system performance metric; e.g., user satisfaction has been measured via surveys which ask users to rate systems during use along dimensions such as task ease, speech in- put/output quality, user expectations and expertise, and user future use (M¨oller, 2005b; Walker et al., 2002; Bonneau-Maynard et al., 2000; Walker et al., 2000; Shriberg et al., 1992). However, it is expen- sive to run experiments over large numbers of users to obtain reliable system performance measures. The PARADISE model (Walker et al., 1997) pro- poses instead to predict system performance, using parameters representing interaction costs and bene- fits between system and user, including task success, dialogue efficiency, and dialogue quality. More for- mally, a set of interaction parameters are measured in a spoken dialogue system corpus, then used in a multivariate linear regression to predict the target performance variable. The resulting model is de- scribed by the formula below, where there are n in- teraction parameters, pi, each weighted by the anal- ysis with a coefficient, wi, which will be negative or positive, depending on whether the model treats pi as a cost or benefit, respectively. The model can then be used to estimate performance during …

Content truncated to 500 words