Assessing and Improving the Performance of Speech Recognition for Incremental Systems

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 380–388, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Assessing and Improving the Performance of Speech Recognition for Incremental Systems Timo Baumann, Michaela Atterer, David Schlangen Institut f¨ur Linguistik Universit¨at Potsdam Potsdam, Germany {timo,atterer,das}@ling.uni-potsdam.de Abstract In incremental spoken dialogue systems, par- tial hypotheses about what was said are re- quired even while the utterance is still ongo- ing. We define measures for evaluating the quality of incremental ASR components with respect to the relative correctness of the par- tial hypotheses compared to hypotheses that can optimize over the complete input, the tim- ing of hypothesis formation relative to the por- tion of the input they are about, and hypothesis stability, defined as the number of times they are revised. We show that simple incremen- tal post-processing can improve stability dra- matically, at the cost of timeliness (from 90 % of edits of hypotheses being spurious down to 10 % at a lag of 320 ms). The measures are not independent, and we show how system de- signers can find a desired operating point for their ASR. To our knowledge, we are the first to suggest and examine a variety of measures for assessing incremental ASR and improve performance on this basis. 1 Introduction Incrementality, that is, the property of beginning to process input before it is complete, is often seen as a desirable property of dialogue systems (e.g., Allen et al. (2001)), as it allows the system to (a) fold processing time (of modules such as parsers, or di- alogue managers) into the time taken by the utter- ance, and (b) react to partial results, for example by generating back-channel utterances or speculatively initiating potentially relevant database queries. Input to a spoken dialogue system normally passes an automatic speech recognizer (ASR) as a first processing module, thus the module’s incre- mentality determines the level of incrementality that can be reached by the system as a whole. Using an ASR system incrementally poses interesting chal- lenges, however. Typically, ASRs use dynamic pro- gramming and the maximum likelihood hypothesis to find the word sequence with the lowest expected likelihood of the sequence containing errors (sen- tence error). Due to the dynamic programming ap- proach, what is considered the best hypothesis about a given stretch of the input signal can change during the recognition process, as more right context which can be used as evidence becomes available. In this paper, we argue that normally used met- rics for ASR evaluation such as word error rate must be complemented with metrics specifically designed for measuring incremental performance, and offer some such metrics. We show that there are various subproperties that are not independent of each other, and that trade-offs are involved if either of those is to be optimized. Finally, we propose ways to im- prove incremental performance (as measured by our metrics) through the use of smoothing techniques. To our knowledge, incremental evaluation met- rics of …

Content truncated to 500 words