Evaluating Subtitle Segmentation for End-to-end Generation Systems

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3069–3078 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3069 Evaluating Subtitle Segmentation for End-to-end Generation Systems Alina Karakanta1,2, Franc¸ois Buet3, Mauro Cettolo1, Franc¸ois Yvon3 1 Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento - Italy 2 University of Trento, Italy 3 Universit´e Paris-Saclay, CNRS, LISN, France {akarakanta,cettolo}@fbk.eu, {buet,yvon}@limsi.fr Abstract Subtitles appear on screen as short pieces of text, segmented based on formal constraints (length) and syntactic/semantic crite- ria. Subtitle segmentation can be evaluated with sequence segmentation metrics against a human reference. However, standard segmentation metrics cannot be applied when systems generate outputs different than the reference, e.g. with end-to-end subtitling systems. In this paper, we study ways to conduct reference-based evaluations of segmentation accuracy irrespective of the textual content. We first conduct a systematic analysis of existing metrics for evaluating subtitle segmentation. We then introduce Sigma, a new Subtitle Segmentation Score derived from an approximate upper-bound of BLEU on segmentation boundaries, which allows us to disentangle the effect of good segmentation from text quality. To compare Sigma with existing metrics, we further propose a boundary projection method from imperfect hypotheses to the true reference. Results show that all metrics are able to reward high quality output but for similar outputs system ranking depends on each metric’s sensitivity to error type. Our thorough analyses suggest Sigma is a promising segmentation candidate but its reliability over other segmentation metrics remains to be validated through correlations with human judgements. Keywords: Subtitling, Segmentation, Evaluation, Metric 1. Introduction Accessibility of audiovisual content has become a le- gal obligation for major TV channels in many coun- tries (EU, 2010) and is also a strong suggestion for uploaded web content (EU, 2016). Subtitles are a means for providing accessibility services, either with intralingual closed captioning for the deaf and hard-of- hearing or with interlingual subtitling in various lan- guages for persons without knowledge of the source language speech. Subtitles are also useful for online talks and educational content and facilitate the compre- hension of speech by language learners. Automatising the generation of subtitles has been a long-standing is- sue (Piperidis et al., 2004; Melero et al., 2006; Volk et al., 2010), and is nowadays more and more often per- formed with neural models trained end-to-end (Lakew et al., 2019; Liu et al., 2020). Automatic generation of subtitles is a difficult task, since subtitles should not only reflect the spoken con- tent, but should also satisfy multiple formal require- ments related to their position on screen, the text length, size and colour, their display duration and synchronization with speech, etc. Additionally, a good segmentation of the transcribed or translated text into subtitles must satisfy syntactic and semantic con- straints, since a segmentation which respects linguistic units has been shown to facilitate comprehension and to lead to more readable subtitles (Perego, 2008; Ra- jendran et al., 2013). Subtitling must thus go well be- yond the automatic transcription and translation of …

Content truncated to 500 words