Joint Generation of Captions and Subtitles with Dual Decoding

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 74 - 82 May 26-27, 2022 c⃝2022 Association for Computational Linguistics Joint Generation of Captions and Subtitles with Dual Decoding Jitao Xu† François Buet† Josep Crego‡ Elise Bertin-Lemée‡ François Yvon† †Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France ‡SYSTRAN, 5 rue Feydeau, 75002 Paris, France {firstname.lastname}@{†limsi.fr,‡systrangroup.com} Abstract As the amount of audio-visual content in- creases, the need to develop automatic cap- tioning and subtitling solutions to match the expectations of a growing international au- dience appears as the only viable way to boost throughput and lower the related post- production costs. Automatic captioning and subtitling often need to be tightly intertwined to achieve an appropriate level of consistency and synchronization with each other and with the video signal. In this work, we assess a dual decoding scheme to achieve a strong cou- pling between these two tasks and show how adequacy and consistency are increased, with virtually no additional cost in terms of model size and training complexity. 1 Introduction As the amount of online audio-visual content con- tinues to grow, the need for captions and subtitles1 in multiple languages also steadily increases, as it widens the potential audience of these contents. T S T S C T S C T C,S a. b. c. d. Figure 1: A graphical view of various captioning and subtitling strategies. T refers to transcripts. C and S respectively denote captions and subtitles. 1We use ‘caption’ to refer to a text written in the same language as the audio and ‘subtitle’ when translated into an- other language. Captions, which are often meant for viewers with hearing difficulties, and subtitles, which are produced for viewers with an imperfect command of the source language, may have slightly different traits, that we ignore here. Both activities are closely related: human sub- title translators often generate subtitles directly based on the original captions without viewing or listening to the original audio/video file. This strat- egy however runs the risk of amplifying, in the subtitle approximations, simplifications or errors present in the captioning. It may even happen that both texts need to be simultaneously displayed on screen: for instance, in countries with several offi- cial languages, or to help foreign language learners. This means that captions and subtitles need to be consistent not only with the video content, but also with each other. It also implies that they should be synchronized (Karakanta et al., 2021). Finally, even in scenarios where only subtitles would be needed, generating captions at the same time may still help to better check the correctness of subtitles. Early approaches to automatic subtitling (e.g. Piperidis et al., 2004) also assumed a pipeline ar- chitecture (Figure 1 (b)), where subtitles are trans- lated from captions derived from automatic speech transcripts. A recent alternative (Figure 1 (a)), which mitigates cascading errors, is to indepen- dently perform captioning and subtitling in an end- to-end manner (Liu et al., 2020; Karakanta et al., 2020a); the risk however is …

Content truncated to 500 words