FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task

Proceedings of the 18th International Conference on Spoken Language Translation, pages 131–137 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 131 FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task Yun Tang* Hongyu Gong* Xian Li Changhan Wang Juan Pino Holger Schwenk Facebook AI ResearchNaman Goyal {yuntang,hygong,xianl,changhan,juancarabina,schwenk,naman}@fb.com Abstract In this paper, we describe our end-to-end multilingual speech translation system sub- mitted to the IWSLT 2021 evaluation cam- paign on the Multilingual Speech Translation shared task. Our system is built by leverag- ing transfer learning across modalities, tasks and languages. First, we leverage general- purpose multilingual modules pretrained with large amounts of unlabelled and labelled data. We further enable knowledge transfer from the text task to the speech task by training two tasks jointly. Finally, our multilingual model is finetuned on speech translation task- specific data to achieve the best translation results. Experimental results show our sys- tem outperforms the reported systems, includ- ing both end-to-end and cascaded based ap- proaches, by a large margin. In some trans- lation directions, our speech translation results evaluated on the public Multilingual TEDx test set are even comparable with the ones from a strong text-to-text translation system, which uses the oracle speech transcripts as input. 1 Introduction Multilingual speech translation (Inaguma et al., 2019) enables translation from audio to text in multiple language directions with a single model. Similar to multilingual text translation, it is sam- ple efficient as the model supports more languages. Furthermore, multilingual speech models can facil- itate positive transfer across languages by learning a common representation space from speech inputs, typically either raw audio or filterbank features. In this paper, we provide a description of our submission to the first multilingual speech trans- lation task at IWSLT 2021. The task evaluates *Yun Tang and Hongyu Gong have equal contribution to this work. speech translations from Spanish (es), French (fr), Portuguese (pt) and Italian (it) into English (en) and Spanish (es). Among them, three translation directions (it-en, it-es and pt-es) are considered zero-shot with respect to the constrained track. In addition, participants are encouraged to submit transcriptions for the relevant languages. Our team, FAIR Speech Translation (FST), par- ticipated in the unconstrained track, where we sub- mitted one primary system and four contrastive systems. We are interested in exploring the effec- tiveness of building a general-purpose multilingual multi-modality model. We leverage large amounts of data, including unlabelled and labelled data from different modalities, to alleviate the data scarcity issue. We build the multilingual model to perform speech translation and speech recognition tasks for all evaluation directions. Our model leverages self- supervised pretraining on both the encoder and the decoder. The model is further improved by knowl- edge transferring from the text-to-text translation task to the speech-to-text translation task under the multitask learning framework. Finally, we finetune the model on parallel speech translation corpora as well as weakly aligned speech translation data through data mining to achieve the best result. In section 2, we described data …

Content truncated to 500 words