CEASR: A Corpus for Evaluating Automatic Speech Recognition

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6477–6485 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6477 CEASR: A Corpus for Evaluating Automatic Speech Recognition Malgorzata Anna Ulasik, Manuela H¨urlimann, Fabian Germann, Esin Gedik, Fernando Benites, Mark Cieliebak Zurich University of Applied Sciences, Winterthur, Switzerland, SpinningBytes AG, Winterthur, Switzerland {ulas, hueu, gedi, benf, ciel}@zhaw.ch, fabian@germann.cc Abstract In this paper, we present CEASR, a Corpus for Evaluating the quality of Automatic Speech Recognition (ASR). It is a data set based on public speech corpora, containing metadata along with transcripts generated by several modern state-of-the-art ASR systems. CEASR provides this data in a unified structure, consistent across all corpora and systems, with normalised transcript texts and metadata. We use CEASR to evaluate the quality of ASR systems by calculating an average Word Error Rate (WER) per corpus, per system and per corpus-system pair. Our experiments show a substantial difference in accuracy between commercial versus open-source ASR tools as well as differences up to a factor ten for single systems on different corpora. Using CEASR allowed us to very efficiently and easily obtain these results. Our corpus enables researchers to perform ASR-related evaluations and various in-depth analyses with noticeably reduced effort, i.e. without the need to collect, process and transcribe the speech data themselves. Keywords: automatic speech recognition, evaluation, speech corpus, ASR systems 1. Introduction Automatic Speech Recognition (ASR) has seen rapid progress over the past decades, fuelled by increasing de- mand for the applications it enables, such as in-meeting as- sistants, voice search, subtitling, virtual speech assistants, intelligent devices, or dictation tools. Closely linked with the development of high-quality ASR systems is the task of ASR quality evaluation, which is essential for both research and practical applications. The standard metric for mea- suring ASR performance is the Word Error Rate (WER), which counts the word-level insertions, deletions and sub- stitutions in a generated transcript (hypothesis) as compared to a gold standard transcription (reference). For commercial ASR providers, it is common practice to advertise a single WER figure for their system, without mentioning the properties of the audio and speech on which this score was obtained. Similarly, open source models are often evaluated on a very small set of reference cor- pora, which gives a limited insight into their capability. The Corpus for Evaluating Automatic Speech Recognition (CEASR1) fills this gap by providing transcriptions of nine English and six German speech corpora that were generated by seven different ASR systems. The current version of CEASR (1.0) is a snapshot of state-of-the-art ASR technol- ogy from 2019. It can serve as a benchmark for assessing and comparing the quality of various ASR systems as well as tracking the progress in ASR technology over time.2 The speech corpora selected for CEASR are standard cor- pora often cited in the literature. They represent a vari- ety of speaking styles (read-aloud vs. spontaneous, mono- logue vs. dialogue), speaker demographics (native vs. non- native, different dialectal regions, …

Content truncated to 500 words