CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7873–7887 July 9-14, 2023 ©2023 Association for Computational Linguistics CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation Yan Zhou1, Qingkai Fang1,2, Yang Feng1,2∗ 1Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2University of Chinese Academy of Sciences, Beijing, China 2001zhouyan@gmail.com, {fangqingkai21b, fengyang}@ict.ac.cn Abstract End-to-end speech translation (ST) is the task of translating speech signals in the source lan- guage into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine transla- tion (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport (CMOT) to overcome the modality gap. We find the alignment be- tween speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation direc- tions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps al- leviate the modality gap between speech and text. 1 1 Introduction Speech translation (ST) is the task to translate speech signals in one language into text in another. Conventional ST systems work in a cascaded mode (STENTIFORD and STEER, 1988; Waibel et al., 1991), by combining automatic speech recognition (ASR) and machine translation (MT), which may suffer from error propagation and high latency. To solve these problems, end-to-end ST is proposed (Bérard et al., 2016; Duong et al., 2016). End-to- end ST directly generates the target text directly from the source audio without the intermediate transcript, which has drawn more attention in re- cent years (Vila et al., 2018; Salesky et al., 2019; Di Gangi et al., 2019b,c; Inaguma et al., 2020; ∗Corresponding author: Yang Feng. 1Code is publicly available at https://github.com/ ictnlp/CMOT. Wang et al., 2020a; Zhao et al., 2021a; Dinh et al., 2022; Duquenne et al., 2022a). End-to-end ST has to perform cross-lingual and cross-modal transformation simultaneously, hence bringing more challenges than ASR or MT. Unfor- tunately, it is harder to collect parallel speech-to- text pairs than text-to-text pairs, which makes ST suffer from the under-fitting problem. To solve the data shortage problem, some researchers propose methods to introduce text-to-text parallel data of MT into ST, including pretraining (Alinejad and Sarkar, 2020; Zheng et al., 2021; Xu et al., 2021), multi-task learning (Le et al., 2020; Vydana et al., 2021; Tang et al., 2021), knowledge distillation (Liu et al., 2019; Gaido et al., 2020; Inaguma et al., 2021), etc. In this way, the knowledge of MT can be transferred to ST, helping learn a better cross- lingual transformation. Although these methods of transfer learning have shown improvements in the translation qual- ity, ST is joint learning of cross-lingual …

Content truncated to 500 words