Differences in User Responses to a Wizard-of-Oz versus Automated System

Proceedings of NAACL-HLT 2013, pages 796–801, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Differences in User Responses to a Wizard-of-Oz versus Automated System Jesse Thomason University of Pittsburgh Pittsburgh, PA 15260, USA jdt34@pitt.edu Diane Litman University of Pittsburgh Pittsburgh, PA 15260, USA litman@cs.pitt.edu Abstract Wizard-of-Oz experimental setup in a dia- logue system is commonly used to gather data for informing an automated version of that system. Previous work has exposed depen- dencies between user behavior towards sys- tems and user belief about whether the sys- tem is automated or human-controlled. This work examines whether user behavior changes when user belief is held constant and the sys- tem’s operator is varied. We perform a post- hoc experiment using generalizable prosodic and lexical features of user responses to a dia- logue system backed with and without a hu- man wizard. Our results suggest that user responses are different when communicating with a wizarded and an automated system, in- dicating that wizard data may be less reliable for informing automated systems than gener- ally assumed. 1 Introduction In a Wizard-of-Oz (WOZ) experimental setup, some or all of the automated portions of a dialogue sys- tem are replaced with a hidden, human evaluator. This setup is often used to gather data from users who believe they are interacting with an automated system (Wolska et al., 2004; Andrews et al., 2008; Becker et al., 2011). This data can inform a down- stream, real automated system. A WOZ experimen- tal protocol calls for holding “all other input and out- put . . . constant so that the only unknown variable is who does the internal processing” (Paek, 2001). Thus, hiding the human wizard’s input by layers of system interface can render that system believably automated. An assumption of this WOZ data-gathering strat- egy is that user behavior will not vary substantially between the WOZ and automated (AUT) experimen- tal setups. However, it was shown in a dialogue sys- tem that training with a small set of data from an au- tomated system gave rise to better performance than training with a large set of data from an analogous wizarded system (Drummond and Litman, 2011). There, it was suggested that differences in system automation may be responsible for the performance gap. It is possible that user responses to these dia- logue systems differed substantially. This paper aims to investigate this possibility by comparing data between a wizarded and automated version of a tutoring dialogue system. We hypoth- esize that what users say and how they say it will differ when the only change is whether the system’s speech recognition and correctness evaluation com- ponents are wizarded or automated. 2 Dialogue System The data for this study is provided by the baseline conditions (one wizarded (WOZ) and one automated (AUT)) of two prior experiments with a spoken tu- torial dialogue system. Users of this system were students recruited at our university, and each was a native speaker of American English. Users were novices and were …

Content truncated to 500 words