Human-Machine Problem Solving Using Spoken Language Systems (SLS): Factors Affecting Performance and User Satisfaction

Human-Machine Problem Solving Using Spoken Language Systems (SLS): Factors Affecting Performance and User Satisfaction Elizabeth Shriberg 1,3, Elizabeth Wade 2'3, Patti Price 3 1University of California at Berkeley, Department of Psychology, Berkeley, CA 94720 2Stanford University, Department of Psychology, Stanford, CA 94305 3SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94303 ABSTRACT We have analyzed three factors affecting user satisfaction and system performance using an SLS implemented in the ATIS domain. We have found that: (I) trade-offs between speed and accuracy have different implications for user sat- isfaction; (2) recognition performance improves over time, at least in part because of a reduction in sentence perplex- ity; and (3) hyperarticulation increases recognition errors, and while instructions can reduce this behavior, they do not result in improved recognition performance. We conclude that while users may adapt to some aspects of an SLS, cer- tain types of user behavior may require technological solu- tions. 1. INTRODUCTION Data collection is a critical component of the DARPA Spo- ken Language Systems (SLS) program. Data are crucial not only for system training, development and evaluation, but also for analyses that can provide insight to guide future research and development. By observing users interacting with an SLS under different conditions, we can assess which issues may best be addressed by human factors and which will require technological solutions. System devel- opers can benefit from considering not only initial use of an SLS, but also the experience of a user over time. Systems based on current technology work best when speech and language closely resemble the training data used to develop the system. However, there is considerable variability in the degree to which the speech and language of new users match that of the training data. The current paper examines the importance of this initial match. It is possible that users whose speech does not conform to the system may be able to adapt their behavior over time (e.g., Stem and Rudnicky [11]). In order to evaluate technology in terms of the demands of the application, we need to understand the extent and the nature of such adaptation and the conditions that affect it. Although system performance can be measured in a number of ways, in this paper, we focus on (1) self-reports of user satisfaction, and (2) recog- nition performance. Further studies could include addi- tional measures. SRI has been collecting data in the air travel planning domain using a number of different systems (see Bly et al. [1]; Kowtko and Price [5]). In moving from wizard-based data collection to the use of SRI's SLS, we observed changes in user behavior that were associated with system errors. Some of these behaviors were adaptive; for exam- ple, learning to avoid out-of-vocabulary words or unusual syntax should facilitate successful interaction. Other behaviors, however, were non-adaptive and could actually impede the interaction. For example, speaking more loudly or in a hyperarticulate style may be detrimental to system performance insofar as these styles differ from those observed in training material dominated …

Content truncated to 500 words