Learning about Voice Search for Spoken Dialogue Systems

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 840–848, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Learning about Voice Search for Spoken Dialogue Systems Rebecca J. Passonneau1, Susan L. Epstein2,3, Tiziana Ligorio2, Joshua B. Gordon4, Pravin Bhutada4 1Center for Computational Learning Systems, Columbia University 2Department of Computer Science, Hunter College of The City University of New York 3Department of Computer Science, The Graduate Center of The City University of New York 4Department of Computer Science, Columbia University becky@cs.columbia.edu, susan.epstein@hunter.cuny.edu, tligorio@gc.cuny.edu, joshua@cs.columbia.edu, pravin.bhutada@gmail.com Abstract In a Wizard-of-Oz experiment with multiple wizard subjects, each wizard viewed automated speech recognition (ASR) results for utterances whose interpretation is critical to task success: requests for books by title from a library data- base. To avoid non-understandings, the wizard directly queried the application database with the ASR hypothesis (voice search). To learn how to avoid misunderstandings, we investi- gated how wizards dealt with uncertainty in voice search results. Wizards were quite suc- cessful at selecting the correct title from query results that included a match. The most suc- cessful wizard could also tell when the query results did not contain the requested title. Our learned models of the best wizard’s behavior combine features available to wizards with some that are not, such as recognition confi- dence and acoustic model scores. 1 Introduction Wizard-of-Oz (WOz) studies have long been used for spoken dialogue system design. In a relatively new variant, a subject (the wizard) is presented with real or simulated automated speech recogni- tion (ASR) to observe how people deal with incor- rect speech recognition output (Rieser, Kruijff- Korbayová, & Lemon, 2005; Skantze, 2003; Stuttle, Williams, & Young, 2004; Williams & Young, 2003, 2004; Zollo, 1999). In these experi- ments, when a wizard could not interpret the ASR output (non-understanding), she rarely asked users to repeat themselves. Instead, the wizard found other ways to continue the task. This paper describes an experiment that pre- sented wizards with ASR results for utterances whose interpretation is critical to task success: re- quests for books from a library database, identified by title. To avoid non-understandings, wizards used voice search (Wang et al., 2008): they direct- ly queried the application database with ASR out- put. To investigate how to avoid errors in understanding (misunderstandings), we examined how wizards dealt with uncertainty in voice search results. When the voice search results included the requested title, all seven of our wizards were likely to identify it. One wizard, however, recognized far better than the others when the voice search results did not contain the requested title. The experiment employed a novel design that made it possible to include system features in models of wizard beha- vior. The principal result is that our learned models of the best wizard’s behavior combine features that are available to wizards with some that are not, such as recognition confidence and acoustic model scores. The next section of the paper motivates our ex- periment. Subsequent sections describe …

Content truncated to 500 words