Language Adapters for Large-Scale MT: The GMU System for the WMT 2022 Large-Scale Machine Translation Evaluation for African Languages Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 154 - 164 December 7-8, 2022 ©2022 Association for Computational Linguistics Continuous Rating as Reliable Human Evaluation of Simultaneous Speech Translation D´avid Javorsk´y Dominik Mach´aˇcek Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {surname}@ufal.mff.cuni.cz Ondˇrej Bojar Abstract Simultaneous speech translation (SST) can be evaluated on simulated online events where hu- man evaluators watch subtitled videos and con- tinuously express their satisfaction by pressing buttons (so called Continuous Rating). Continu- ous Rating is easy to collect, but little is known about its reliability, or relation to comprehen- sion of foreign language document by SST users. In this paper, we contrast Continuous Rating with factual questionnaires on judges with different levels of source language knowl- edge. Our results show that Continuous Rating is easy and reliable SST quality assessment if the judges have at least limited knowledge of the source language. Our study indicates users’ preferences on subtitle layout and presentation style and, most importantly, provides a signifi- cant evidence that users with advanced source language knowledge prefer low latency over fewer re-translations. 1 Introduction Simultaneous speech translation (SST) is a technol- ogy that assists users to understand and follow a speech in a foreign language in real-time. The users may need such an assistance because of limited knowledge of the source language, the speaker’s non-native accent, or the topic and vocabulary. The technology can be used for the target languages, for which human interpretation is unavailable, e.g. due to capacity reasons. Candidate systems for simultaneous speech translation differ in quality of translation, latency and the approach to stability. Some are streaming, only adding more words (Grissom II et al., 2014; Gu et al., 2017; Arivazhagan et al., 2019; Press and Smith, 2018; Xiong et al., 2019; Ma et al., 2019; Zheng et al., 2019; Iranzo Sanchez et al., 2022), some allow re-translation as more input arrives (M¨uller et al., 2016b; Niehues et al., 2016; Dess- loch et al., 2018; Niehues et al., 2018; Arivazhagan et al., 2020). Finally, subtitle presentation options Figure 1: A detail of the default layout with the video document “Dinge Erkl¨art: Impfen...”.1 The video is at the top, overlaid by two lines of subtitles in Czech, followed by buttons for Continuous Rating. The button labels are: 1: Worse; 2: Average; 3: Good; 0: I do not understand at all. (size of subtitling window, layout, allowed reading time, font size, etc.) also affect users’ impression. The combination of the re-translating approach and limited space for subtitles is challenging because of “flicker”, i.e. the updates to the text that the user is reading at the moment, has already read, or that has been scrolled away. The subtitling options impact the amount of flicker, reading comfort and delay and may affect the general usability. The evaluation of the traditional, text-to-text machine translation (MT) has been researched for many years (see e.g. Han, 2018 or develop- ments and discussion within the series of WMT, Akhbardeh et …

Content truncated to 500 words