WER we are and WER we think we are

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3290–3295 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3290 WER we are and WER we think we are Piotr Szyma´nski1,2,∗, Piotr ˙Zelasko3,∗, Mikołaj Morzy1,4, Adrian Szymczak1, Marzena ˙Zyła-Hoppe1, Joanna Banaszczak1, Łukasz Augustyniak1,3 Jan Mizgajski1,4, Yishay Carmiel1 1 Avaya Inc., USA 2 Wrocław University of Science and Technology, Wrocław, Poland 3 Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA 4 Pozna´n University of Technology, Pozna´n, Poland ∗ Equal contribution piotr.szymanski@pwr.edu.pl, pzelasko@jhu.edu Abstract Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commer- cial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB’05 public benchmark. We show that WERs are significantly higher than the best re- ported results. We formulate a set of guide- lines which may aid in the creation of real-life, multi-domain datasets with high quality anno- tations for training and testing of robust ASR systems. 1 Introduction The last few years have witnessed unprecedented progress in Automatic Speech Recognition (ASR) systems. They have become ubiquitous in every- day lives, from phones voice assistants through dictation of text messages and e-mails to managing household appliances with home assistants. It is not surprising that major vendors are trying to show- case the quality and accuracy of their products. A comprehensive benchmark of available ASRs (Syn- naeve, 2020) cites word error rates (WERs) as low as 2%–3% on standard datasets. These reports may incur a false conviction that automatic speech recognition is mostly a solved problem. Nothing could be further from the truth. One possible cause for this misconception and gross over-estimation of the accuracy of contem- porary ASRs is the confounding of two regimes of speech recognition. The majority of interac- tions with ASRs happen in the context of chatbot- like interactions, when a human is fully aware of speaking to a machine. In these circumstances most people simplify their utterances, speaking in short, well-structured phrases which obey the correct grammatical structure of either an inter- rogative or an imperative sentence. Siegert and Kr¨uger (2018) point out that conversations between people are more lively and dynamic, while the conversations with Alexa remain mostly simple request/response interactions. Significant acous- tic differences allowed them to build a clasifier which achieves 81% of accuracy for distinguising between human-human and human-chatbot interac- tions. Human-chatbot interactions are in stark con- trast with typical spontaneous human-human con- versations. Such conversations are riddled with var- ious disfluencies (discourse markers, filled pauses, back-channeling), lack clear phrase borders, and human utterances are often not terminated correctly. Oftentimes information is exchanged using non- verbal means, for example, prosody, paralanguage, and non-verbal vocalizations. Hill et al. (2015) found that human–chatbot communication lacked much of the richness of vocabulary …

Content truncated to 500 words