Multilingual Speech Translation from Efficient Finetuning of Pretrained Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 827–838 August 1–6, 2021. ©2021 Association for Computational Linguistics 827 Multilingual Speech Translation from Efficient Finetuning of Pretrained Models Xian Li*, Changhan Wang*, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli Facebook AI {xianl,changhan,yuntang,chau,yuqtang,juancarabina, abaevski,aconneau,michaelauli}@fb.com Abstract We present a simple yet effective approach to build multilingual speech-to-text (ST) transla- tion through efficient transfer learning from a pretrained speech encoder and text de- coder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross- modality transfer ability by only finetuning 10 ∼ 50% of the pretrained parameters. This effectively leverages large pretrained models at low training cost such as wav2vec 2.0 for acoustic modeling, and mBART for multilin- gual text generation. This sets a new state-of- the-art for 36 translation directions (and sur- passing cascaded ST for 30 of them) on the large-scale multilingual ST benchmark CoV- oST 2 (Wang et al., 2020b) (+6.4 BLEU on average for En-X directions and +6.7 BLEU for X-En directions). Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model (+5.6 BLEU on average across 28 directions), mak- ing it an appealing approach for attaining high- quality speech translation with improved pa- rameter and data efficiency. 1 Introduction Recent advances in pretraining over unlabeled data and then finetuning on labeled data leads to sig- nificant performance improvement in text under- standing and generation tasks (Devlin et al., 2019; Liu et al., 2020; Conneau et al., 2019; Radford, 2018). Lately, such text pretraining and finetuning paradigms have been extended to other modalities: audio (Schneider et al., 2019; Baevski et al., 2020), images (Su et al., 2019; Lu et al., 2019), and video (Sun et al., 2019). At the same time, pretraining and finetuning techniques have improved multi- tasking applications significantly, such as multi- lingual translation, cross-lingual representations, question-answering and so on (Raffel et al., 2020; Self Attention FFN LayerNorm LayerNorm x Length Adaptor wav2vec 2.0 Self Attention Encoder Attention FFN LayerNorm LayerNorm LayerNorm x mBART Finetune LayerNorm and Attention Pretrained Modules Encoder Decoder Figure 1: An overview of the proposed speech-to-text translation via transfer learning from efficient finetun- ing of single-modality pretrained models. The pro- posed LNA finetuning is applied to each layer. Yang et al., 2019; Tang et al., 2020). In this paper, we advance the one-model-for-all paradigm further by adapting audio and multilingual text pretraining and finetuning to improve multilingual speech-to- text translation. Our contributions are as follows: • We propose a simple and effective approach to combine pretrained single-modality modules to perform speech-to-text translation. With minimal architecture change, we add a cross- modal adaptor to bridge the length discrep- ancy between audio encoder output and text decoder input. Our approach can also perform multi-task finetuning with both speech-to-text translation and text-to-text translation tasks where we find joint training with the latter brings further gains. • …

Content truncated to 500 words