T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5794 - 5806 December 7-11, 2022 ©2022 Association for Computational Linguistics T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation Paul-Ambroise Duquenne Meta AI & Inria padqn@fb.com Hongyu Gong Meta AI hygong@fb.com Benoît Sagot Inria benoit.sagot@inria.fr Holger Schwenk Meta AI schwenk@fb.com Abstract We present a new approach to perform zero- shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size repre- sentation space. Then, we compare different ap- proaches to decode these multimodal and mul- tilingual fixed-size representations, enabling zero-shot translation between languages and modalities. All our models are trained with- out the need of cross-modal labeled translation data. Despite a fixed-size representation, we achieve very competitive results on several text and speech translation tasks. In particular, we outperform the state of the art for zero-shot speech translation on Must-C. We also intro- duce the first results for zero-shot direct speech- to-speech and text-to-speech translation. 1 Introduction Most, if not all, current state-of-the-art text and speech translation systems are based on a sequence- to-sequence approach and an attention mechanism to connect the encoder and decoder. Such mod- els require labeled data to be trained end-to-end. For text-to-text (T2T) translation, this labeled data, called bitexts, is available in large amounts for a number of language pairs, in particular since large-scale bitext mining initiatives like ParaCrawl (Bañón et al., 2020) and CCMatrix (Schwenk et al., 2021). Finding training data for speech-to-text (S2T) translation is more challenging, but several data collection efforts exist, like mTEDx (Salesky et al., 2021), CoVoST (Wang et al., 2020a,b), and Must-C (Di Gangi et al., 2019). Finally, speech-to- speech (S2S) translation suffers from scarcity of end-to-end labeled data and current S2S systems are limited to a very small number of language pairs. Very recent works start to consider mining labeled data for S2S, e.g. (Duquenne et al., 2021). Unsupervised representation learning is very suc- cessfully used to initialize the encoder and/or de- coder of a sequence-to-sequence model, thereby Figure 1: Summary of the model architecture. lowering the amount of labeled data needed to train or fine-tune the model end-to-end. Approaches in- clude for instance XLM (Conneau and Lample, 2019), XLSR (Conneau et al., 2020), wav2vec (Baevski et al., 2020), data2vec (Baevski et al., 2022) and mSLAM (Bapna et al., 2022). In this work, we propose a new modular architec- ture for text and speech translation, which is based on a common fixed-size multilingual and mul- timodal internal representation, and encoders and decoders which are independently trained. We explore several variants of teacher-student train- ing to learn text and speech encoders for multiple languages, which are compatible with the embed- ding space of the LASER encoder (Artetxe and Schwenk, 2019). In contrast to preceding works on multilingual and multimodal representations, we also train text decoders for multiple languages which are able to generate translations given the joint representation. Finally, we demonstrate that it is possible to train …

Content truncated to 500 words