STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7050 - 7062 May 22-27, 2022 c⃝2022 Association for Computational Linguistics STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation Qingkai Fang1,2†, Rong Ye3, Lei Li4*†, Yang Feng1,2*, Mingxuan Wang3* 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences, Beijing, China 3 ByteDance AI Lab 4 University of California, Santa Barbara {fangqingkai21b,fengyang}@ict.ac.cn {yerong,wangmingxuan.89}@bytedance.com, leili@cs.ucsb.edu Abstract How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine transla- tion (MT) capabilities to ST, but neglect the representation discrepancy across modalities. In this paper, we propose the Speech-TExt Manifold Mixup (STEMM) method to cali- brate such discrepancy. Specifically, we mix up the representation sequences of different modalities, and take both unimodal speech se- quences and multimodal mixed sequences as input to the translation model in parallel, and regularize their output predictions with a self- learning framework. Experiments on MuST- C speech translation benchmark and further analysis show that our method effectively alle- viates the cross-modal representation discrep- ancy, and achieves significant improvements over a strong baseline on eight translation di- rections. 1 Introduction Speech-to-text translation (ST) aims at translating acoustic speech signals into text in a foreign lan- guage, which has wide applications including voice assistants, translation for multinational video con- ferences, and so on. Traditional ST methods usu- ally combine automatic speech recognition (ASR) and machine translation (MT) in a cascaded man- ner (Sperber et al., 2017; Cheng et al., 2018; Sper- ber et al., 2019; Dong et al., 2019b; Zhang et al., 2019a; Lam et al., 2021b), which might suffer from error propagation and high latency. To break this bottleneck, end-to-end ST systems attracted much * indicates corresponding authors. † Work was done while at ByteDance AI Lab. Part of joint project between ICT/CAS and ByteDance AI Lab. Work was done when QF was a member of the joint project. Code and models are publicly available at https:// github.com/ictnlp/STEMM. “he” “on” “road” “bicycles” he on road bicycles “the” the “rode” rode “he” “bicycles” “road” “rode” on rode bicycles road “the” he “on” the “speech” text “speech” text Figure 1: STEMM aims at bridging the modality gap of speech and text. Different modalities with the same meaning are projected to a shared space. attention recently (Wang et al., 2020b,c; Dong et al., 2021a,b; Han et al., 2021; Inaguma et al., 2021a; Tang et al., 2021a), which learn a unified model to generate translations from speech directly. Some recent work has shown great potential for end-to- end speech translation, even surpassing traditional cascaded systems (Ye et al., 2021; Xu et al., 2021). As a cross-modal task, a major challenge in train- ing an end-to-end ST model is the representation discrepancy across modalities, which means there is a modality gap between speech representations and text embeddings, as shown in …

Content truncated to 500 words