Direct Speech-to-Speech Translation With Discrete Units

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3327 - 3339 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Direct Speech-to-Speech Translation With Discrete Units Ann Lee1, Peng-Jen Chen1, Changhan Wang1, Jiatao Gu1, Sravya Popuri1, Xutai Ma1,2, Adam Polyak1, Yossi Adi1, Qing He1, Yun Tang1, Juan Pino1, Wei-Ning Hsu1 Meta AI1 Johns Hopkins University2 {annl,wnhsu}@fb.com Abstract We present a direct speech-to-speech transla- tion (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text genera- tion. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then training a sequence- to-sequence speech-to-unit translation (S2UT) model to predict the discrete representations of the target speech. When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass. Experiments on the Fisher Spanish- English dataset show that the proposed frame- work yields improvement of 6.7 BLEU com- pared with a baseline direct S2ST model that predicts spectrogram features. When trained without any text transcripts, our model perfor- mance is comparable to models that predict spectrograms and are trained with text super- vision, showing the potential of our system for translation between unwritten languages1. 1 Introduction Speech translation aims at converting speech from one language into speech or text in another lan- guage. The technology helps bridge the commu- nication barriers between people speaking differ- ent languages and can provide access to multime- dia content in different languages. Conventional speech-to-text translation (S2T) systems take a cas- caded approach by concatenating automatic speech recognition (ASR) and machine translation (MT). In recent years, end-to-end S2T (Bérard et al., 2016) is proposed to alleviate the error propagation issue between ASR and MT. These S2T models can be further combined with text-to-speech (TTS) 1Audio samples are available at //facebookresearch.github.io/speech_https: translation/direct_s2st_units/index.html synthesis to provide both speech and text transla- tion, which allows the technology to be adopted in a wider range of applications. More recently, researchers have started exploring building direct speech-to-speech translation (S2ST) models without relying on text generation as an in- termediate step (Jia et al., 2019b, 2021). Direct S2ST has the benefits of lower computational costs and inference latency as fewer decoding steps are needed compared to cascaded systems. In addi- tion, direct S2ST is a natural approach for sup- porting translation for languages without a writing system (Tjandra et al., 2019; Zhang et al., 2020). Jia et al. (2019b) first addresses the problem by training an attention-based sequence-to-sequence model that maps source speech spectrograms into target spectrograms. Model training is challenging as it requires the model to learn not only the align- ment between two languages but also the acoustic and linguistic characteristics of both languages. As a result, there is a performance gap between the direct S2ST system and an S2T+TTS cascaded sys- tem. The recent success in self-supervised …

Content truncated to 500 words