Comparing Objective and Subjective Measures of Usability in a Human-Robot Dialogue System

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 879–887, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Comparing Objective and Subjective Measures of Usability in a Human-Robot Dialogue System Mary Ellen Foster and Manuel Giuliani and Alois Knoll Informatik VI: Robotics and Embedded Systems Technische Universit¨at M¨unchen Boltzmannstraße 3, 85748 Garching bei M¨unchen, Germany {foster,giuliani,knoll}@in.tum.de Abstract We present a human-robot dialogue sys- tem that enables a robot to work together with a human user to build wooden con- struction toys. We then describe a study in which na¨ıve subjects interacted with this system under a range of conditions and then completed a user-satisfaction ques- tionnaire. The results of this study pro- vide a wide range of subjective and ob- jective measures of the quality of the in- teractions. To assess which aspects of the interaction had the greatest impact on the users’ opinions of the system, we used a method based on the PARADISE evalua- tion framework (Walker et al., 1997) to de- rive a performance function from our data. The major contributors to user satisfac- tion were the number of repetition requests (which had a negative effect on satisfac- tion), the dialogue length, and the users’ recall of the system instructions (both of which contributed positively). 1 Introduction Evaluating the usability of a spoken language dia- logue system generally requires a large-scale user study, which can be a time-consuming process both for the experimenters and for the experimen- tal subjects. In fact, it can be difficult even to define what the criteria are for evaluating such a system (cf. Novick, 1997). In recent years, tech- niques have been introduced that are designed to predict user satisfaction based on more easily mea- sured properties of an interaction such as dialogue length and speech-recognition error rate. The de- sign of such performance methods for evaluating dialogue systems is still an area of open research. The PARADISE framework (PARAdigm for DIalogue System Evaluation; Walker et al. (1997)) describes a method for using data to derive a per- formance function that predicts user-satisfaction scores from the results on other, more easily com- puted measures. PARADISE uses stepwise mul- tiple linear regression to model user satisfaction based on measures representing the performance dimensions of task success, dialogue quality, and dialogue efficiency, and has been applied to a wide range of systems (e.g., Walker et al., 2000; Litman and Pan, 2002; M¨oller et al., 2008). If the result- ing performance function can be shown to predict user satisfaction as a function of other, more eas- ily measured system properties, it will be widely applicable: in addition to making it possible to evaluate systems based on automatically available data from log files without the need for extensive experiments with users, for example, such a per- formance function can be used in an online, incre- mental manner to adapt system behaviour to avoid entering a state that is likely to reduce user satis- faction, or can be …

Content truncated to 500 words