Cross-modal Contrastive Learning for Speech Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5099 - 5113 July 10-15, 2022 ©2022 Association for Computational Linguistics Cross-modal Contrastive Learning for Speech Translation Rong Ye 1, Mingxuan Wang 1, Lei Li 2∗ 1 ByteDance AI Lab, 2 University of California, Santa Barbara {yerong, wangmingxuan.89}@bytedance.com leili@cs.ucsb.edu Abstract How can we learn unified representations for spoken utterances and their written text? Learning similar representations for seman- tically similar speech and text is important for speech translation. To this end, we pro- pose ConST, a cross-modal contrastive learn- ing method for end-to-end speech-to-text trans- lation. We evaluate ConST and a variety of previous baselines on a popular benchmark MuST-C. Experiments show that the proposed ConST consistently outperforms the previous methods, and achieves an average BLEU of 29.4. The analysis further verifies that ConST indeed closes the representation gap of dif- ferent modalities — its learned representation improves the accuracy of cross-modal speech- text retrieval from 4% to 88%. Code and models are available at https://github. com/ReneeYe/ConST. 1 Introduction End-to-end speech-to-text translation (E2E ST) be- comes important in many internet products and real applications. An E2E ST system accepts au- dio signals as the input and generates the target translation using a single model. Compared with the conventional cascade ST models, E2E ST mod- els have achieved almost comparable (Bentivogli et al., 2021) or even superior (Ansari et al., 2020; Potapczyk and Przybysz, 2020; Xu et al., 2021) performance. The performance of an E2E ST model is still re- stricted for languages with relatively small parallel data, compared to text machine translation (MT). Existing approaches for ST focus on using addi- tional data from MT and automatic speech recog- nition (ASR). This can be realized through pre- training approaches (Zheng et al., 2021; Dong et al., 2021b,a) or multi-task training frameworks (Tang et al., 2021b; Ye et al., 2021; Han et al., 2021). ∗Partial work was done while at ByteDance. “Speech” Transcript “It is a nice day!” “What are you going to do today?” It is a nice day! What are you going to do today? “It's a new day full of energy.” It's a new day full of energy. “if you take chances” if you take chances (a) Current “It is a nice day!” “What are you going to do today?” It is a nice day! What are you going to do today? “It's a new day full of energy.” It's a new day full of energy. “if you take chances” if you take chances “Speech” Transcript (b) Ideal Figure 1: Illustration of representations for speech and transcript text (projected to 2D). (a) representations learned by existing models. Pairs of speech and text representations are distant. (b) an ideal representation that we expect, where different modalities with same meaning should stay close to each other. Different from the data perspective, this paper investigates the bottleneck of E2E ST from the neural representation perspective. We believe that when the representation …

Content truncated to 500 words