End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021

Proceedings of the 18th International Conference on Spoken Language Translation, pages 110–119 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 110 End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021 Gerard I. G´allego, Ioannis Tsiamas, Carlos Escolano, Jos´e A. R. Fonollosa, Marta R. Costa-juss`a TALP Research Center, Universitat Polit`ecnica de Catalunya, Barcelona {gerard.ion.gallego,ioannis.tsiamas,carlos.escolano jose.fonollosa,marta.ruiz}@upc.edu Abstract This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech transla- tion system, which combines pre-trained mod- els (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the sys- tem and pre-training it, can increase the con- vergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuST- C test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom seg- mentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untran- scribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation. 1 Introduction Typically, a speech translation (ST) system is com- posed of an automatic speech recognition (ASR) and a machine translation (MT) model, which is known as cascade system. However, in recent years, end-to-end models have gained popularity within the research community. These systems are encoder-decoder architectures capable of di- rectly translating speech without intermediate sym- bolic representations. This approach solves classi- cal shortcomings of cascade ST systems, e.g. the error propagation or the slow inference time (Weiss et al., 2017). Nevertheless, while there are plenty of data available to train ASR and MT systems, there are not as many datasets for ST, despite some recent efforts (Di Gangi et al., 2019a; Wang et al., 2020b). Moreover, this approach is inherently more difficult because the encoder has to perform both acoustic modeling and semantic encoding. For these reasons, end-to-end ST systems still struggle to achieve the performance of cascade ST mod- els. Still, last year’s IWSLT was the first time an end-to-end system had the best performance in the evaluation campaign (Potapczyk and Przybysz, 2020; Ansari et al., 2020). Hence, given the in- creasing interest in end-to-end ST systems, and the potential gains from advancing research on them, we decided to focus on developing such a system for this year’s offline task. When there are not enough data for a task, a com- mon practice is to use pre-trained components, like BERT (Devlin et al., 2019) for various NLP tasks. In the ST field, the idea …

Content truncated to 500 words