Extending the MuST-C Corpus for a Comparative Evaluation of Speech Translation Technology

Extending the MuST-C Corpus for a Comparative Evaluation of Speech Translation Technology Luisa Bentivogli1, Mauro Cettolo1, Marco Gaido1,2, Alina Karakanta1,2, Matteo Negri1, Marco Turchi1 1Fondazione Bruno Kessler 2University of Trento {bentivo,cettolo,mgaido,akarakanta,negri,turchi}@fbk.eu Abstract This project aimed at extending the test sets of the MuST-C speech translation (ST) corpus with new reference transla- tions. The new references were collected from professional post-editors working on the output of different ST systems for three language directions: English– German/Italian/Spanish. In this paper, we describe how the data were collected and how they are distributed. As an evi- dence of their usefulness, we also sum- marize the findings of the first compara- tive evaluation of cascade and direct ST approaches, which was carried out rely- ing on the collected data. The project was partially funded by the European Asso- ciation for Machine Translation (EAMT) through its 2020 Sponsorship of Activities programme. 1 Project overview In this project we created and released additional reference translations for the test sets of the MuST- C corpus (Cattoni et al., 2021). The new references were collected for three language directions, i.e. En–De/Es/It, and consist of professional post-edits of the output of two state-of-the-art systems that represent the main current ST approaches, namely a cascade and a direct system. Data. Our evaluation data are drawn from MuST-C, which is the largest freely available mul- tilingual corpus for ST. It is based on English TED talks and currently covers 14 language directions, © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. with English audio segments automatically aligned with their corresponding manual transcripts and translations. In MuST-C, a Common Test Set in- cludes segments from talks that are common in all directions, thus making it possible to evaluate and compare systems across languages. For the three language directions addressed in the project, this common section includes the same 27 TED talks, for a total of around 2,500 largely overlapping seg- ments.1 For all language directions, we selected from MuST-C Common the same English audio portions from each talk, in order to obtain repre- sentative groups of contiguous segments that are comparable across languages. Furthermore, to en- sure high data quality, we manually checked the selected samples and kept only those segments for which the audio-transcript-translation alignment was correct. Each of the 3 resulting post-editing test sets – henceforth PE sets – contains 550 seg- ments, corresponding to ∼10,000 English source words. Then, we translated the PE sets with two ST systems. One represents the traditional cascade approach, in which the task is performed by means of a pipeline of separate automatic speech recogni- tion (ASR) and machine translation (MT) compo- nents. The other adopts the more recent direct ap- proach, which relies on a single encoder–decoder architecture that directly translates the source au- dio signal bypassing intermediate representations. Post-editing. To prepare the data for the two post-editing (PE) tasks, we followed the main criteria adopted in the IWSLT PE-based evalua- tion campaigns …

Content truncated to 500 words