A Semi-Automated Live Interlingual Communication Workflow Featuring Intralingual Respeaking: Evaluation and Benchmarking

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4405–4413 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4405 A Semi-Automated Live Interlingual Communication Workflow Featuring Intralingual Respeaking: Evaluation and Benchmarking Tomasz Korybski, Elena Davitti, Constantin Orăsan, Sabine Braun University of Surrey, Centre for Translation Studies {t.korybski, e.davitti, c.orasan, s.braun}@surrey.ac.uk Abstract In this paper, we present a semi-automated workflow for live interlingual speech-to-text communication which seeks to reduce the shortcomings of existing ASR systems: a human respeaker works with a speaker-dependent speech recognition software (e.g., Dragon Naturally Speaking) to deliver punctuated same-language output of superior quality than obtained using out-of-the-box automatic speech recognition of the original speech. This is fed into a machine translation engine (the EU’s eTranslation) to produce live-caption ready text. We benchmark the quality of the output against the output of best-in-class (human) simultaneous interpreters working with the same source speeches from plenary sessions of the European Parliament. To evaluate the accuracy and facilitate the comparison between the two types of output, we use a tailored annotation approach based on the NTR model (Romero-Fresco and Pöchhacker, 2017). We find that the semi-automated workflow combining intralingual respeaking and machine translation is capable of generating outputs that are similar in terms of accuracy and completeness to the outputs produced in the benchmarking workflow, although the small scale of our experiment requires caution in interpreting this result. Keywords: interpreting, respeaking, Automatic Speech Recognition, Machine Translation 1. Introduction Traditionally, live interlingual communication has been achieved only with the help of human interpreters. In recent years, advances in Automatic Speech Recognition (ASR) and Machine Translation (MT) have enabled researchers and industrial actors to explore fully automated speech-to- speech (STS) and speech-to-text (STT) workflows without human input in the delivery of live communication across languages (for example Microsoft’s Skype translator). However, although such workflows deliver increasingly promising results when tested on specific datasets (such as call centre data or audiobook data), they encounter challenges when faced with real-life spoken source input characterised by large vocabularies, ambient noise, babble, and articulation variability, which may hamper error-free ASR (El Hannani, 2021). In this paper, we present a semi-automated workflow for live interlingual STT which seeks to reduce the errors introduced in a STT pipeline by the ASR stage in cascaded systems: a professional human respeaker works with speaker-dependent speech recognition software (in our case Dragon Naturally Speaking1) to deliver segmented and punctuated same-language output of superior quality than obtained using out-of-the-box ASR of the original speech. This output is later fed into an MT engine to produce live caption-ready text. Because in our experiments we used speeches from the plenary sessions of the European Parliament (EP), we applied the eTranslation engine, an online MT service provided by the European Commission. We benchmark the quality of the output against the output of best-in-class (human) simultaneous interpreters working with the same source speeches. The evaluation approach is based on the NTR model (Romero-Fresco and Pöchhacker, 2017). 1 https://www.nuance.com/en-gb/dragon.html …

Content truncated to 500 words