Between Flexibility and Consistency: Joint Generation of Captions and Subtitles

Proceedings of the 18th International Conference on Spoken Language Translation, pages 215–225 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 215 Between Flexibility and Consistency: Joint Generation of Captions and Subtitles Alina Karakanta1,2, Marco Gaido1,2, Matteo Negri1, Marco Turchi1 1 Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento - Italy 2 University of Trento, Italy akarakanta,mgaido,negri,turchi}@fbk.eu Abstract Speech translation (ST) has lately received growing interest for the generation of subtitles without the need for an intermediate source language transcription and timing (i.e. cap- tions). However, the joint generation of source captions and target subtitles does not only bring potential output quality advantages when the two decoding processes inform each other, but it is also often required in multilingual sce- narios. In this work, we focus on ST models which generate consistent captions-subtitles in terms of structure and lexical content. We further introduce new metrics for evaluating subtitling consistency. Our findings show that joint decoding leads to increased performance and consistency between the generated cap- tions and subtitles while still allowing for suf- ficient flexibility to produce subtitles conform- ing to language-specific needs and norms. 1 Introduction New trends in media localisation call for the rapid generation of subtitles for vast amounts of au- diovisual content. Speech translation, and espe- cially direct approaches (B´erard et al., 2016; Bahar et al., 2019), have recently shown promising results with high efficiency because they do not require a transcription (manual or automatic) of the source speech but generate the target language subtitles directly from the audio. However, obtaining the intralingual subtitles (hereafter “captions”) is nec- essary for a range of applications, while in some settings captions need to be displayed along with the target language subtitles. Such “bilingual sub- titles” are useful in multilingual online meetings, in countries with multiple official languages, or for language learners and audiences with different accessibility needs. In those cases, captions and subtitles should not only be consistent with the visual and acoustic dimension of the audiovisual material but also between each other, for example in the number of blocks (pieces of time-aligned text) they occupy, their length and segmentation. Consistency is vital for user experience, for ex- ample in order to elicit the same reaction among multilingual audiences, or to facilitate the quality assurance process in the localisation industry. Previous work in ST for subtitling has focused on generating interlingual subtitles (Matusov et al., 2019; Karakanta et al., 2020a), a) without consid- ering the necessity of obtaining captions consistent with the target subtitles, and b) without examining whether the joint generation leads to improvements in quality. We hypothesise that knowledge shar- ing between the tasks of transcription and transla- tion could lead to such improvements. Moreover, joint generation with a single system can avoid the maintenance of two different models, increase efficiency, and in turn speed up the localisation process. Lastly, if joint generation improves con- sistency, joint models could increase automation in subtitling applications where consistency is a desideratum. In this work, we address …

Content truncated to 500 words