MT for subtitling: User evaluation of post-editing productivity

MT for subtitling: User evaluation of post-editing productivity Maarit Koponen ∗ Umut Sulubacak ∗ Kaisa Vitikainen †∗ J¨org Tiedemann ∗ ∗ University of Helsinki † Yle {name.surname}@helsinki.fi {name.surname}@yle.fi Abstract This paper presents a user evaluation of machine translation and post-editing for TV subtitles. Based on a process study where 12 professional subtitlers translated and post-edited subtitles, we compare ef- fort in terms of task time and number of keystrokes. We also discuss examples of specific subtitling features like condensa- tion, and how these features may have af- fected the post-editing results. In addi- tion to overall MT quality, segmentation and timing of the subtitles are found to be important issues to be addressed in future work. 1 Introduction Developments in machine translation (MT) in the last two decades have led to significant improve- ments in translation quality. The success and popu- larity of statistical machine translation (SMT) sys- tems were matched and eventually surpassed by neural machine translation (NMT). As quality has improved, the use of MT and post-editing (PE) has also increased in professional translation work- flows. Broadly, PE refers to the practice of using MT output as a raw version checked and corrected by the translator. The use of MT and PE has been found to increase productivity in various trans- lation scenarios (e.g. Plitt and Masselot, 2010). However, this workflow appears less common in the field of audiovisual translation (AVT). For ex- ample, Bywood et al. (2017) note that while spe- cialised subtitling software with various function- c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. alities are used, technologies like translation mem- ory (TM) or MT have not been widely adopted in AVT. Matusov et al. (2019) suggest that a reason for the lower rate of MT adoption in the AVT field may be that current NMT systems are not suited for the particular features of subtitle translation. This paper presents a pilot study carried out in November 2019 examining how the use of MT and PE in the subtitling workflow affects the work and productivity of subtitlers. In the study, 12 professional subtitle translators worked on a series of tasks in four language pairs (Finnish→English, Finnish→Swedish, English→Finnish, and Swedish→Finnish). They created interlingual (translated) subtitles for short video clips both with and without MT output. To assess productivity and effort, keylogging data were recorded during these tasks. Task time and technical effort represented by keystrokes were compared between post-editing and translation from scratch. We first discuss related work on MT for subti- tling and approaches to user evaluation of MTPE in Section 2. The MT models and subtitle align- ment are presented in Section 3. Section 4 outlines the user data collection, and Section 5 presents the analysis of productivity measures. Section 6 dis- cusses observations on PE changes, followed by future work and conclusions. 2 Related work 2.1 Machine translation for subtitling Interlingual translated subtitles are a solution (along with dubbing and voice-overs) for …

Content truncated to 500 words