Generating Synthetic Speech from SpokenVocab for Speech Translation

Findings of the Association for Computational Linguistics: EACL 2023, pages 1975–1981 May 2-6, 2023 ©2023 Association for Computational Linguistics Generating Synthetic Speech from SpokenVocab for Speech Translation Jinming Zhao Gholamreza Haffari Ehsan Shareghi Department of Data Science & AI, Monash University firstname.lastname@monash.edu Abstract Training end-to-end speech translation (ST) systems requires sufficiently large-scale data, which is unavailable for most language pairs and domains. One practical solution to the data scarcity issue is to convert text-based machine translation (MT) data to ST data via text-to- speech (TTS) systems.Yet, using TTS systems can be tedious and slow. In this work, we pro- pose SpokenVocab, a simple, scalable and ef- fective data augmentation technique to convert MT data to ST data on-the-fly. The idea is to retrieve and stitch audio snippets, correspond- ing to words in an MT sentence, from a spoken vocabulary bank. Our experiments on multiple language pairs show that stitched speech helps to improve translation quality by an average of 1.83 BLEU score, while performing equally well as TTS-generated speech in improving translation quality. We also showcase how Spo- kenVocab can be applied in code-switching ST for which often no TTS systems exit.1 1 Introduction End-to-end (E2E) speech-to-text translation (ST) models require large amounts of data to train (Sper- ber and Paulik, 2020). Despite the emerging ST datasets (Cattoni et al., 2021; Wang et al., 2021), their size is considerably smaller compared to text-based machine translation (MT) data. A common remedy to tackle the data scarcity is- sue is to leverage text-based MT data in train- ing ST systems. Common approaches include multi-task learning (Anastasopoulos and Chiang, 2018; Ye et al., 2021), transfer learning & pretrain- ing (Bansal et al., 2019; Wang et al., 2020) and knowledge distillation (Inaguma et al., 2021; Tang et al., 2021). A more straightforward alternative is to convert text-based MT data to ST via text-to-speech (TTS) synthesis engines (Pino et al., 2019; Jia et al., 2019). 1Our code is available at https://github.com/ mingzi151/SpokenVocab Figure 1: Overview of generating synthetic speech from SpokenVocab on-the-fly. The first step is to prepare the SpokenVocab bank offline and the second step is to retrieve and stitch audio snippets from the bank by words in a sentence. This method is less commonly used despite its sim- plicity and effectiveness,2 mainly due to practical reasons: (i) TTS models have slow inference time and may incur monetary costs; (ii) the conversion is required for each MT datasets. Recently, Lam et al. (2022) proposed to generate synthetic speech with- out using TTS models. However, their approach is based on real ST data, and thus cannot be extended to MT data. In this work, we propose a simple, effective and efficient data augmentation approach to convert MT data to ST data on-the-fly. The idea is to pre- pare a set of spoken words, forming a spoken vo- cabulary (SpokenVocab) bank, and then generate synthetic speech by retrieving and stitching spoken 2Only one work out of 8 uses TTS to augment data in the IWSLT2022 offline …

Content truncated to 500 words