Simple and Effective Unsupervised Speech Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10771–10784 July 9-14, 2023 ©2023 Association for Computational Linguistics Simple and Effective Unsupervised Speech Translation Changhan Wang, Hirofumi Inaguma, Peng-Jen Chen, Ilia Kulikov, Yun Tang Wei-Ning Hsu, Michael Auli, Juan Pino Meta - Fundamental AI Research (FAIR) {changhan,hirofumii,pipibjc,kulikov,yuntang, wnhsu,michaelauli,juancarabina}@meta.com Abstract The amount of labeled data to train models for speech tasks is limited for most languages, how- ever, the data scarcity is exacerbated for speech translation which requires labeled data cover- ing two different languages. To address this issue, we study a simple and effective approach to build speech translation systems without la- beled data by leveraging recent advances in un- supervised speech recognition, machine trans- lation and speech synthesis, either in a pipeline approach, or to generate pseudo-labels for train- ing end-to-end speech translation models. Fur- thermore, we present an unsupervised domain adaptation technique for pre-trained speech models which improves the performance of downstream unsupervised speech recognition, especially for low-resource settings. Experi- ments show that unsupervised speech-to-text translation outperforms the previous unsuper- vised state of the art by 3.2 BLEU on the Libri- Trans benchmark, on CoVoST 2, our best sys- tems outperform the best supervised end-to-end models (without pre-training) from only two years ago by an average of 5.0 BLEU over five X-En directions. We also report competitive results on MuST-C and CVSS benchmarks. 1 Introduction Training supervised speech systems requires large amounts of labeled data which is often not avail- able for all but a small fraction of the over 7,000 languages spoken around the world (Lewis et al., 2022). Despite much recent effort in creating speech translation corpora (Di Gangi et al., 2019a; Wang et al., 2021b), only a few dozen language directions are covered. The lack of labeled train- ing data is even more acute for speech translation because it requires aligned labeled data in two lan- guages which increases the effort to create such datasets. This poses the question of whether speech translation systems can be built using less labeled data or no labeled data at all. Recent work on unsupervised speech recognition has achieved performance that can enable useful systems using no labeled data (Yeh et al., 2019; Liu et al., 2018; Chen et al., 2019; Baevski et al., 2021; Liu et al., 2022a), enabled in large part by the advances in self-supervised speech represen- tation learning (Schneider et al., 2019; Baevski et al., 2020). These techniques were also used to build unsupervised text-to-speech systems (Liu et al., 2022b). Similarly, unsupervised text-to-text machine translation has shown great promise for certain language directions (Conneau et al., 2018; Lample et al., 2018; Artetxe et al., 2018). In this paper, we study a method to build end- to-end unsupervised speech-to-text and speech-to- speech translation systems trained on synthetic training data obtained by cascading existing un- supervised techniques: we first transcribe speech utterances in the source language using unsuper- vised speech recognition (Baevski et al., 2021; Liu et al., 2022a), …

Content truncated to 500 words