Towards the evaluation of automatic simultaneous speech translation from a communicative perspective

Proceedings of the 18th International Conference on Spoken Language Translation, pages 245–254 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 245 Towards the evaluation of automatic simultaneous speech translation from a communicative perspective Claudio Fantinuoli Mainz University/KUDO fantinuoli@uni-mainz.de Bianca Prandi Mainz University prandi@uni-mainz.de Abstract In recent years, automatic speech-to-speech and speech-to-text translation has gained mo- mentum thanks to advances in artificial intel- ligence, especially in the domains of speech recognition and machine translation. The qual- ity of such applications is commonly tested with automatic metrics, such as BLEU, primar- ily with the goal of assessing improvements of releases or in the context of evaluation cam- paigns. However, little is known about how the output of such systems is perceived by end users or how they compare to human perfor- mances in similar communicative tasks. In this paper, we present the results of an ex- periment aimed at evaluating the quality of a real-time speech translation engine by compar- ing it to the performance of professional si- multaneous interpreters. To do so, we adopt a framework developed for the assessment of human interpreters and use it to perform a man- ual evaluation on both human and machine performances. In our sample, we found bet- ter performance for the human interpreters in terms of intelligibility, while the machine per- forms slightly better in terms of informative- ness. The limitations of the study and the pos- sible enhancements of the chosen framework are discussed. Despite its intrinsic limitations, the use of this framework represents a first step towards a user-centric and communication- oriented methodology for evaluating real-time automatic speech translation. 1 Introduction Real-time or simultaneous speech translation (ST) aims at translating a continuous speech input from one language to another with the lowest latency1 and highest quality possible. In recent years, auto- matic speech translation systems have been devel- 1In this context, we broadly define latency as the time delay from when an utterance is pronounced in the source language to when it gets translated in the target language. oped at scale, and their quality has improved sig- nificantly (Sperber and Paulik, 2020). At present, research is increasingly focusing on end-to-end trainable encoder-decoder models, i.e. speech-to- speech (STS) or speech-to-text (STT) translation systems that directly match source and target lan- guage (Di Gangi et al., 2018; Jia et al., 2019; Ansari et al., 2020). Nonetheless, the cascading approach is de facto still the mainstream solution for speech translation (ST). The main reason is that this ap- proach benefits from the remarkable improvements in automatic speech recognition (ASR) (Chiu et al., 2018) and machine translation (MT) (Barrault et al., 2020) obtained thanks to the wealth of task-specific data available. In cascading systems, the process of translating from speech to text or from speech to speech is performed by a series of concatenated modules. In most cases, these systems apply ASR to the speech input, and then pass the results on to an MT engine. Since a short latency is an important …

Content truncated to 500 words