UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15655–15680 July 9-14, 2023 ©2023 Association for Computational Linguistics UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units Hirofumi Inaguma♡, Sravya Popuri♡, Ilia Kulikov♡, Peng-Jen Chen♡, Changhan Wang♡, Yu-An Chung♡, Yun Tang♡, Ann Lee♡, Shinji Watanabe♣, Juan Pino♡ FAIR, Meta AI♡, Carnegie Mellon University♣ {hirofumii,juancarabina}@meta.com Abstract Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST archi- tecture, UnitY, which first generates textual rep- resentations and predicts discrete acoustic units subsequently. We enhance the model perfor- mance by subword prediction in the first-pass decoder, advanced two-pass decoder architec- ture design and search strategy, and better train- ing regularization. To leverage large amounts of unlabeled text data, we pre-train the first- pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83× decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, pre- dicting discrete units achieves 2.51× decoding speed-up compared to that case. 1 Introduction Automatic speech translation to another language is an indispensable technology for international communications, with the spread of social media and virtual communications nowadays. A tradi- tional approach of speech-to-speech translation (S2ST) is to cascade automatic speech recogni- tion (ASR), machine translation (MT), and text- to-speech (TTS) components, each of which is op- timized separately on different data (Lavie et al., 1997; Nakamura et al., 2006; Wahlster, 2013). With the emergence of sequence-to-sequence mod- els (Sutskever et al., 2014; Cho et al., 2014; Bah- danau et al., 2015), however, it is getting prevail- ing to adopt a direct approach1. This approach 1Lee et al. (2022a) defines a direct S2ST model as a model that does not use intermediate text representations while Jia consists in translating input speech into the other language based on a single architecture with fewer components than the cascaded systems (Jia et al., 2019b; Tjandra et al., 2019; Zhang et al., 2021). The direct approach is attractive for building a low- latency system with a simplified pipeline, thus re- ducing developing costs. However, direct S2ST models suffer from poor performance due to data scarcity, similar to direct speech-to-text transla- tion (S2TT) models (Bérard et al., 2016). In the field of S2TT, data shortage has been addressed by leveraging pre-training (Bérard et al., 2018; Wang et al., 2021c; Tang et al., 2022), multi-task learn- ing (Weiss et al., 2017; Tang et al., 2021), pseudo labeling (Jia et al., 2019a; Pino et al., 2020), knowl- edge distillation (Liu et al., 2019; Inaguma et al., 2021b). Consequently, the translation quality of di- rect S2TT models is approaching that of cascaded S2TT models (Ansari et al., 2020; Anastasopoulos et al., 2021). These techniques have also shown the …

Content truncated to 500 words