Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 265 - 276 May 26-27, 2022 c⃝2022 Association for Computational Linguistics Pretrained Speech Encoders and Efficient Fine-tuning Methods for Speech Translation: UPC at IWSLT 2022 Ioannis Tsiamas∗, Gerard I. Gállego∗, Carlos Escolano, José A. R. Fonollosa, Marta R. Costa-jussà TALP Research Center, Universitat Politècnica de Catalunya, Barcelona {ioannis.tsiamas,gerard.ion.gallego,carlos.escolano jose.fonollosa,marta.ruiz}@upc.edu Abstract This paper describes the submissions of the UPC Machine Translation group to the IWSLT 2022 Offline Speech Translation and Speech- to-Speech Translation tracks. The offline task involves translating English speech to German, Japanese and Chinese text. Our Speech Trans- lation systems are trained end-to-end and are based on large pretrained speech and text mod- els. We use an efficient fine-tuning technique that trains only specific layers of our system, and explore the use of adapter modules for the non-trainable layers. We further inves- tigate the suitability of different speech en- coders (wav2vec 2.0, HuBERT) for our mod- els and the impact of knowledge distillation from the Machine Translation model that we use for the decoder (mBART). For segment- ing the IWSLT test sets we fine-tune a pre- trained audio segmentation model and achieve improvements of 5 BLEU compared to the given segmentation. Our best single model uses HuBERT and parallel adapters and achieves 29.42 BLEU at English-German MuST-C tst- COMMON and 26.77 at IWSLT 2020 test. By ensembling many models, we further increase translation quality to 30.83 BLEU and 27.78 accordingly. Furthermore, our submission for English-Japanese achieves 15.85 and English- Chinese obtains 25.63 BLEU on the MuST-C tst-COMMON sets. Finally, we extend our system to perform English-German Speech-to- Speech Translation with a pretrained Text-to- Speech model. 1 Introduction In the last few years, end-to-end (or direct) Speech Translation (ST) models have gained popularity in the research community. These systems differ from the classical cascade ones in their architec- ture, where instead of concatenating an Automatic Speech Recognition (ASR) model and a Machine Translation (MT) system, they directly translate ∗Equal contribution speech into the target language without an inter- mediate transcription. This approach solves some limitations of cascade ST systems, like error propa- gation and slow inference times. But on the other hand, such approaches require more data to be com- petitive, which are not as abundant as ASR and MT data (Sperber and Paulik, 2020). However, the performance gap between the two approaches has become very small in the last years (Bentivogli et al., 2021), with end-to-end approaches having the best performances for the IWSLT 2020 test set in the last two evaluation campaigns (Ansari et al., 2020; Anastasopoulos et al., 2021). Following this research trend, we participate in the Offline Speech Translation task of IWSLT 2022 (Anastasopoulos et al., 2022) with end-to-end sys- tems that are built on top of our last year’s sub- mission (Gállego et al., 2021). The approach we follow is to leverage large pretrained speech and text models, in order to reduce the required amount of data usually needed to train competitive …

Content truncated to 500 words