Understanding and Bridging the Modality Gap for Speech Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15864–15881 July 9-14, 2023 ©2023 Association for Computational Linguistics Understanding and Bridging the Modality Gap for Speech Translation Qingkai Fang1,2, Yang Feng1,2∗ 1Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2University of Chinese Academy of Sciences, Beijing, China {fangqingkai21b, fengyang}@ict.ac.cn Abstract How to achieve better end-to-end speech trans- lation (ST) by leveraging (text) machine trans- lation (MT) data? Among various existing techniques, multi-task learning is one of the effective ways to share knowledge between ST and MT in which additional MT data can help to learn source-to-target mapping. How- ever, due to the differences between speech and text, there is always a gap between ST and MT. In this paper, we first aim to understand this modality gap from the target-side representa- tion differences, and link the modality gap to another well-known problem in neural machine translation: exposure bias. We find that the modality gap is relatively small during train- ing except for some difficult cases, but keeps increasing during inference due to the cascad- ing effect. To address these problems, we propose the Cross-modal Regularization with Scheduled Sampling (CRESS) method. Specif- ically, we regularize the output predictions of ST and MT, whose target-side contexts are de- rived by sampling between ground truth words and self-generated words with a varying proba- bility. Furthermore, we introduce token-level adaptive training which assigns different train- ing weights to target tokens to handle difficult cases with large modality gaps. Experiments and analysis show that our approach effectively bridges the modality gap, and achieves promis- ing results in all eight directions of the MuST-C dataset.1 1 Introduction End-to-end speech translation (ST) aims to trans- late speech signals to text in another language di- rectly. Compared to traditional cascaded meth- ods, which combine automatic speech recogni- tion (ASR) and machine translation (MT) models in a pipeline manner, end-to-end ST could avoid ∗Corresponding author: Yang Feng. 1Code is publicly available at https://github.com/ ictnlp/CRESS. error propagation and high latency (Sperber and Paulik, 2020). Recently, end-to-end ST models have achieved comparable or even better results than cascaded ST models (Bentivogli et al., 2021; Anastasopoulos et al., 2021, 2022). However, due to the scarcity of ST data, it is diffi- cult to directly learn a mapping from source speech to the target text. Previous works often leverage MT data to help the training with multi-task learn- ing (Ye et al., 2022; Tang et al., 2021a). By shar- ing encoder and decoder between ST and MT, the model tends to learn similar representations from different modalities. In this way, the auxiliary MT task can help build the source-to-target mapping. However, there remains a gap between ST and MT due to the differences between speech and text. In this paper, we measure the modality gap with rep- resentation differences of the last decoder layer between ST and MT, because the representation of this layer will be mapped into …

Content truncated to 500 words