Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 159–168 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023 Sara Papi3,2, Marco Gaido3, Matteo Negri3 3Fondazione Bruno Kessler 2University of Trento {spapi,mgaido,negri}@fbk.eu Abstract This paper describes the FBK’s participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST sys- tem shows a reduced computational-aware la- tency compared to the one achieved by the top- ranked systems in the 2021 and 2022 rounds of the task, with gains of up to 3.5 BLEU. Our automatic subtitling system outperforms the only-existing solution based on a direct system by 3.7 and 1.7 SubER in English-German and English-Spanish respectively. 1 Introduction In recent years, the advances in natural language processing and machine learning led to a surge of interest in developing speech translation (ST) sys- tems that can translate speech from one language into text in another language without human inter- vention. Significant progress has been specially made toward end-to-end ST models (Bérard et al., 2016; Weiss et al., 2017) trained to directly trans- late speech without the intermediate steps of tran- scription (through automatic speech recognition - ASR) and translation (through machine translation - MT). Along with this growing interest in direct ST, also accompanied by a reduction of the perfor- mance gap with respect to cascaded architectures (Bentivogli et al., 2021), other trends have emerged thanks to deep learning advancements, which made it possible to deploy direct solutions to perform the task in real-time (i.e. to produce partial translations while continuing to process the input audio) or to automatically generate subtitles for audiovisual content (i.e. pieces of translated text which have to conform to specific spatiotemporal constraints and be synchronized with the video). The International Workshop on Spoken Lan- guage Translation (IWSLT) is playing an impor- tant role in advancing the state-of-the-art in these fields by organizing a series of evaluation cam- paigns (Ansari et al., 2020; Anastasopoulos et al., 2021, 2022) focused on simultaneous speech trans- lation (SimulST) and, this year for the first time, automatic subtitling. These campaigns provide a unique opportunity for researchers to compare their systems against others, share their findings, and identify areas for further improvement. In this paper, we describe FBK’s participation in the IWSLT 2023 Evaluation Campaigns (Agar- wal et al., 2023) for simultaneous translation and automatic subtitling. Motivated by the promising results reported in previous works (Ren et al., 2020; Papi et al., 2022a), our approach is characterized by the use of direct ST models to address both …

Content truncated to 500 words