Towards a methodology for evaluating automatic subtitling

Towards a methodology for evaluating automatic subtitling Alina Karakanta1,2, Luisa Bentivogli1, Mauro Cettolo1, Matteo Negri1, Marco Turchi1 1Fondazione Bruno Kessler 2University of Trento {akarakanta,bentivo,cettolo,negri,turchi}@fbk.eu Abstract In response to the growing interest towards automatic subtitling, the 2021 EAMT- funded project “Towards a methodology for evaluating automatic subtitling” aimed at collecting subtitle post-editing data in a real use case scenario where professional subtitlers edit automatically generated sub- titles. The post-editing setting includes, for the first time, automatic generation of timestamps and segmentation, and focuses on the effect of timing and segmentation edits on the post-editing process. The col- lected data will serve as the basis for in- vestigating how subtitlers interact with au- tomatic subtitling and for devising evalua- tion methods geared to the multimodal na- ture and formal requirements of subtitling. 1 Project overview Automatic subtitling is the task of generating tar- get language subtitles for a given video without any intermediate human transcription and timing of the source speech. The source speech in the video is automatically transcribed, translated and segmented into subtitles, which are synchronised with the speech – a process called automatic spot- ting (or auto-spotting). Automatic subtitling is be- coming a task of increasing interest for the MT community, practitioners and the audiovisual in- dustry. Despite the technological advancements, the evaluation of automatic subtitling still repre- sents a significant research gap. Popular MT eval- uation metrics consider only content-related pa- rameters (translation quality), but not form-related © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. parameters, such as format (length and segmen- tation) and timing (synchronisation with speech, reading speed), which are important features for high-quality subtitles (Carroll and Ivarsson, 1998). Moreover, the way subtitlers interact with auto- matically generated subtitles has not been yet ex- plored, since the majority of works which con- ducted human evaluations of the post-editing effort in MT for subtitling have focused on edits in the textual content (Volk et al., 2010; Bywood et al., 2017; Matusov et al., 2019; Koponen et al., 2020). This project seeks to investigate automatic sub- titling, the factors contributing to post-editing ef- fort and their relation to the quality of the out- put. This is achieved through the collection of rich, product- and process-based subtitling data in a real use case scenario where professional subti- tlers edit automatically translated, spotted and seg- mented subtitles in a dedicated subtitling environ- ment. The richness of the data collected during this one-year project is ideal for understanding the op- erations performed by subtitlers while they inter- act with automatic subtitling in their professional environment and for applying mixed methods ap- •proaches to: Investigate the correlation between amount of text editing, adjustments in auto-spotting and post- •editing temporal/technical effort Explore the effect of auto-spotting edits on the •total post-editing process Investigate the variability in subtitle segmenta- •tion decisions among subtitlers Propose tentative metrics for auto-spotting quality and subtitle segmentation 2 Data collection Three professional subtitlers with experience in post-editing …

Content truncated to 500 words