Why Aren’t We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1746–1761 July 9-14, 2023 ©2023 Association for Computational Linguistics Why Aren’t We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts Piotr Szyma´nski and Łukasz Augustyniak and Adrian Szymczak Wroclaw University of Science and Technology, Poland {piotr.szymanski,lukasz.augustyniak,adrian.szymczak}@pwr.edu.pl Mikołaj Morzy Poznan University of Technology Poland mikolaj.morzy@put.poznan.pl Krzysztof Surdyk Piotr ˙Zelasko Meaning.Team Inc, USA pzelasko@meaning.team Abstract Transcripts of spontaneous human speech present a significant obstacle for traditional NER models. The lack of grammatical struc- ture of spoken utterances and word errors in- troduced by the ASR make downstream NLP tasks challenging. In this paper, we examine in detail the complex relationship between ASR and NER errors which limit the ability of NER models to recover entity mentions from sponta- neous speech transcripts. Using publicly avail- able benchmark datasets (SWNE, Earnings-21, OntoNotes), we present the full taxonomy of ASR-NER errors and measure their true impact on entity recognition. We find that NER models fail to recognize entity spans even if no word errors are introduced by the ASR. We also show why the F1 score is inadequate to evaluate NER models on conversational transcripts1. 1 Introduction The performance of NLP models tends to deterio- rate significantly when the models are applied to the raw outputs of the Automatic Speech Recog- nition (ASR) system. We coin the term ASR-NLP gap to describe this phenomenon. Despite unprece- dented advances in modern language models, the transcript of a spontaneous human-human conver- sation remains an insurmountable challenge for most models. This is particularly true for Named Entity Recognition (NER) models, which struggle to retrieve even the most basic entity mentions from spontaneous speech. be1All code necessary to reproduce our results can found in https://github.com/niedakh/ asr-ner-eval-repository Three primary factors contribute to the existence of the ASR-NLP gap. Firstly, the structure of spon- taneous human conversations is diametrically dif- ferent from the prescriptive written language used to train language models. These models can use the grammatical structure present in the training corpora, such as part-of-speech sequences, depen- dency trees, and dialog acts. On the other hand, spontaneous conversations lack sentence structure. They contain repetitions, back-channeling, phatic expressions, and other artifacts of turn-taking. The second challenge comes from the original ASR output containing neither punctuation nor sentence segmentation. These have to be restored by an auxiliary downstream model. Thus, NLP mod- els trained on prescriptive written text or scripted conversations already have to process the out-of- domain input. The third problem stems from ASR systems injecting word errors into the transcript. Due to efficiency requirements, most ASR systems use unsophisticated language models such as n- gram models with limited vocabulary. Thus, many utterances in the input audio may be unrecognized and deleted from the output, while other utterances may cause substitutions or insertions of erroneous tokens into the output. Consider the following sentence: "I am to see [Dr Smith]PERSON at [9 am]TIME on [Monday, May 14th]DATE". The NER …

Content truncated to 500 words