Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6469–6476 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6469 Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains Kallirroi Georgila, Anton Leuski, Volodymyr Yanov, David Traum Institute for Creative Technologies, University of Southern California 12015 Waterfront Drive, Los Angeles, CA 90094-2536, USA {kgeorgila, leuski, yanov, traum}@ict.usc.edu Abstract We evaluate several publicly available off-the-shelf (commercial and research) automatic speech recognition (ASR) systems across diverse dialogue domains (in US-English). Our evaluation is aimed at non-experts with limited experience in speech recognition. Our goal is not only to compare a variety of ASR systems on several diverse data sets but also to measure how much ASR technology has advanced since our previous large-scale evaluations on the same data sets. Our results show that the performance of each speech recognizer can vary significantly depending on the domain. Furthermore, despite major recent progress in ASR technology, current state-of-the-art speech recognizers perform poorly in domains that require special vocabulary and language models, and under noisy conditions. We expect that our evaluation will prove useful to ASR consumers and dialogue system designers. Keywords: off-the-shelf speech recognizers, dialogue systems, diverse dialogue domains 1. Introduction In this paper we evaluate several publicly available off-the- shelf (commercial and research) automatic speech recog- nition (ASR) systems, using data collected from deployed spoken dialogue systems as well as from human-human conversations in 6 domains (in US-English). A natural language dialogue system usually takes human speech as its input, thus a speech recognizer resides at the very front-end of such a system. Other natural language processing units highly rely on the output of a speech rec- ognizer. Thus a speech recognizer directly affects the over- all system performance. An ASR system mainly uses two models: an acoustic model responsible for modelling the sounds that make up words, and a language model respon- sible for modelling word sequences. The two models work together to find the best hypotheses (word sequences) cor- responding to a given speech signal. Every dialogue system has a target user population. For ex- ample, a conversational assistant, such as Amazon Alexa, Apple Siri, Google Home, or Microsoft Cortana, handles various questions in a broad domain while a dialogue sys- tem developed for military purposes must understand mili- tary terms (see below). Thus there are many factors to con- sider in selecting an ASR system for a particular applica- tion, among them: • The domain and vocabulary that the speech recognizer is expected to handle. • The acoustic environment in which the speech recog- nizer operates. • The time it takes for a speech recognizer to generate an output. There is often a trade-off between the quality of the ASR output and the time it takes to generate that output; real-time dialogue systems may be willing to accept a somewhat degraded output in return for lower latencies. • Whether the speech recognizer can generate incre- mental outputs or waits until the speaker …

Content truncated to 500 words