Textless Speech-to-Speech Translation on Real Data

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 860 - 872 July 10-15, 2022 ©2022 Association for Computational Linguistics Textless Speech-to-Speech Translation on Real Data Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, Wei-Ning Hsu Meta AI {annl,wnhsu}@fb.com Abstract We present a textless speech-to-speech trans- lation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi- speaker target speech and train the systems with real-world S2ST data. The key to our approach is a self-supervised unit-based speech normal- ization technique, which finetunes a pre-trained speech encoder with paired audios from multi- ple speakers and a single reference speaker to reduce the variations due to accents, while pre- serving the lexical content. With only 10 min- utes of paired data for speech normalization, we obtain on average 3.2 BLEU gain when train- ing the S2ST model on the VoxPopuli S2ST dataset, compared to a baseline trained on un- normalized speech target. We also incorporate automatically mined S2ST data and show an additional 2.0 BLEU gain. To our knowledge, we are the first to establish a textless S2ST tech- nique that can be trained with real-world data and works for multiple language pairs1. 1 Introduction Speech-to-speech translation (S2ST) technology can help bridge the communication gap between people speaking different languages. Conventional S2ST systems (Lavie et al., 1997; Nakamura et al., 2006) usually rely on a cascaded approach by first translating speech into text in the target language, either with automatic speech recognition (ASR) fol- lowed by machine tranlsation (MT), or an end-to- end speech-to-text translation (S2T) model (Bérard et al., 2016), and then applying text-to-speech (TTS) synthesis to generate speech output. On the other hand, researchers have started ex- ploring direct S2ST (Jia et al., 2019, 2021; Tjandra 1Audio samples are available at //facebookresearch.github.io/speech_https: translation/textless_s2st_real_data/ index.html et al., 2019; Zhang et al., 2020; Kano et al., 2021; Lee et al., 2021), which aims at translating speech in the source language to speech in the target lan- guage without the need of text generation as an intermediate step. However, text transcriptions or phoneme annotations of the speech data is often still needed during model training for multitask learning (Jia et al., 2019; Lee et al., 2021) or for learning a decoder that generates intermediate rep- resentations (Jia et al., 2021; Kano et al., 2021) to facilitate the generation of speech output. More than 40% of the languages in the world are without text writing systems2, while very lim- ited work exist to tackle the challenge of train- ing direct S2ST systems without the use of any text data (Tjandra et al., 2019; Zhang et al., 2020). Moreover, due to the lack of S2ST training data, previous work on direct S2ST mainly rely …

Content truncated to 500 words