WACO: Word-Aligned Contrastive Learning for Speech Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3891–3907 July 9-14, 2023 ©2023 Association for Computational Linguistics WACO: Word-Aligned Contrastive Learning for Speech Translation Siqi Ouyang1, Rong Ye2, Lei Li1 1University of California, Santa Barbara, CA, USA siqiouyang@ucsb.edu, leili@cs.ucsb.edu 2ByteDance AI Lab, Shanghai, China yerong@bytedance.com Abstract End-to-end Speech Translation (E2E ST) aims to directly translate source speech into target text. Existing ST methods perform poorly when only extremely small speech-text data are available for training. We observe that an ST model’s performance closely correlates with its embedding similarity between speech and source transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a simple and effective method for extremely low-resource speech-to-text translation. Our key idea is bridging word-level representations for both speech and text modalities via con- trastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark, and on a low-resource direction Maltese-English from IWSLT 2023. Our exper- iments demonstrate that WACO outperforms the best baseline by 9+ BLEU points with only 1-hour parallel ST data. Code is available at https://github.com/owaski/WACO. 1 Introduction End-to-end speech translation (E2E ST) directly translates speech in a source language to text in a target language, without intermediate pipelines. E2E ST has witnessed significant progress in trans- lation quality (Duong et al., 2016; Weiss et al., 2017; Bérard et al., 2018; Jia et al., 2019; Gaido et al., 2020; Dong et al., 2021a; Ye et al., 2022a) . However, existing E2E ST methods still perform poorly when only a limited amount of parallel ST data are available. How can we build a highly per- formant ST model with extremely low resource, e.g. only 1 hour of parallel data (approximately a few hundred utterances)? We analyze the encoder representations from a directly learned ST model1 and find that its av- erage embeddings from speech and transcript are similar at the sequence level but still not aligned 1We use XSTNet (Ye et al., 2021) as an example. ����� ��� �� ���� �� ������ ���� ����� ����� ������� ���� ��� ������ ���� �� ������ �������� �������� ����������� �� �� � ����� (a) Existing ����� ��� �� ���� ������ ��� ������ ������ ���� ����� ������ ������� ���� ����� �� �������� �������� ����������� �� (b) Ideal Figure 1: Better representations of speech and transcript text (projected to 2D) will lead to improved speech translation. Ideal representations (b) should not only be similar at the sequence level, but also closely align corresponding words between speech and transcript. well on the word level (Figure 1a). An ideal ST model should encode a speech utterance closely aligned with the representations of corresponding words in its transcript text (Figure 1b). Prior meth- ods attempt to use additional parallel data from machine translation (MT) and automatic speech recognition (ASR) to align speech and text repre- sentations. However, most do not explicitly reduce the word level representation gap among speech and text. We hypothesize that such misalignment at the word …

Content truncated to 500 words