CVSS Corpus and Massively Multilingual Speech-to-Speech Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6691–6703 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6691 CVSS Corpus and Massively Multilingual Speech-to-Speech Translation Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, Heiga Zen Google Research jiaye@google.com Abstract We introduce CVSS, a massively multilingual-to-English speech-to-speech translation (S2ST) corpus, covering sentence-level parallel S2ST pairs from 21 languages into English. CVSS is derived from the Common Voice (Ardila et al., 2020) speech corpus and the CoVoST 2 (Wang et al., 2021b) speech-to-text translation (ST) corpus, by synthesizing the translation text from CoVoST 2 into speech using state-of-the-art TTS systems. Two versions of translation speech in English are provided: 1) CVSS-C: All the translation speech is in a single high-quality canonical voice; 2) CVSS-T: The translation speech is in voices transferred from the corresponding source speech. In addition, CVSS provides normalized translation text which matches the pronunciation in the translation speech. On each version of CVSS, we built baseline multilingual direct S2ST models and cascade S2ST models, verifying the effectiveness of the corpus. To build strong cascade S2ST baselines, we trained an ST model on CoVoST 2, which outperforms the previous state-of-the-art trained on the corpus without extra data by 5.8 BLEU. Nevertheless, the performance of the direct S2ST models approaches the strong cascade baselines when trained from scratch, and with only 0.1 or 0.7 BLEU difference on ASR transcribed translation when initialized from matching ST models. Keywords: speech-to-speech translation, speech-to-text translation, multilingual, cross-lingual voice transferring 1. Introduction Speech-to-speech translation (S2ST) is an important means for breaking down the communication barriers between people speaking different languages. Conven- tionally, S2ST systems are built with a cascade of auto- matic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis sub-systems, which are text-centric. Recently, work on S2ST without relying on intermediate text representa- tion are emerging, such as end-to-end direct S2ST (Jia et al., 2019b; Kano et al., 2021; Jia et al., 2021a) and cascade S2ST based on discrete speech representation (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2022; Ma et al., 2021; Lee et al., 2021). However, as of today, publicly available corpora directly suitable for such research are extremely limited (see Table 1). In this paper, we introduce CVSS, a Common Voice- based Speech-to-Speech translation corpus. CVSS is directly derived from the CoVoST 2 ST corpus, which is further derived from the Common Voice speech cor- pus. CVSS provides sentence-level parallel speech-to- speech translation pairs from 21 languages into En- glish, namely from Arabic (ar), Catalan (ca), Welsh (cy), German (de), Estonian (et), Spanish (es), Persian (fa), French (fr), Indonesian (id), Italian (it), Japanese (ja), Latvian (lv), Mongolian (mn), Dutch (nl), Por- tuguese (pt), Russian (ru), Slovenian (sl), Swedish (sv), Tamil (ta), Turkish (tr), and Chinese (zh). The source speech in these 21 languages is crowd-sourced human volunteer recordings from the Common Voice project, totalling 1153 hours. Two versions of transla- tion speech in English are provided …

Content truncated to 500 words