Back Translation for Speech-to-text Translation Without Transcripts

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4567–4587 July 9-14, 2023 ©2023 Association for Computational Linguistics Back Translation for Speech-to-text Translation Without Transcripts Qingkai Fang1,2, Yang Feng1,2∗ 1Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2University of Chinese Academy of Sciences, Beijing, China {fangqingkai21b, fengyang}@ict.ac.cn Abstract The success of end-to-end speech-to-text trans- lation (ST) is often achieved by utilizing source transcripts, e.g., by pre-training with automatic speech recognition (ASR) and machine trans- lation (MT) tasks, or by introducing additional ASR and MT data. Unfortunately, transcripts are only sometimes available since numerous unwritten languages exist worldwide. In this paper, we aim to utilize large amounts of target- side monolingual data to enhance ST without transcripts. Motivated by the remarkable suc- cess of back translation in MT, we develop a back translation algorithm for ST (BT4ST) to synthesize pseudo ST data from monolin- gual target data. To ease the challenges posed by short-to-long generation and one-to-many mapping, we introduce self-supervised discrete units and achieve back translation by cascad- ing a target-to-unit model and a unit-to-speech model. With our synthetic ST data, we achieve an average boost of 2.3 BLEU on MuST-C En→De, En→Fr, and En→Es datasets. More experiments show that our method is especially effective in low-resource scenarios.12 1 Introduction End-to-end speech-to-text translation (ST) means directly translating speech in the source lan- guage to target text without generating source transcripts (Bérard et al., 2016; Duong et al., 2016). Different from traditional cascading meth- ods which first transcribe the speech with automatic speech recognition (ASR) and then translate the transcripts into the target text with machine trans- lation (MT), end-to-end ST has the potential to reduce latency and avoid error propagation. Hence, it has drawn much attention and achieved great suc- ∗Corresponding author: Yang Feng. 1Code is publicly available at https://github.com/ ictnlp/BT4ST. 2Examples of synthetic ST data are available at https: //bt4st.github.io/ and Appendix B. cess in recent years (Anastasopoulos et al., 2021, 2022). However, it is challenging to train an end-to-end ST model with only speech-translation pairs. Tra- ditional cascaded models learn cross-modal map- ping with ASR and cross-lingual mapping with MT. In contrast, end-to-end ST requires simultaneous cross-modal and cross-lingual mapping, which is more complicated and usually relies on more train- ing data. However, the amount of ST data is usually limited due to the high cost of data collection, so the ST model trained with only speech-translation pairs is usually unsatisfactory. To tackle these problems, researchers often uti- lize source transcripts to assist ST training by in- troducing auxiliary ASR and MT tasks. With abun- dant ASR and MT data, the ASR task can help the model learn better cross-modal mapping, while the MT task can help learn better cross-lingual map- ping, which can significantly improve ST as shown in recent ST studies (Wang et al., 2020a; Xu et al., 2021; Ye et al., 2021; Fang et al., 2022). Unfortu- nately, source transcripts …

Content truncated to 500 words