Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5291 - 5302 December 7-11, 2022 ©2022 Association for Computational Linguistics Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation Chen Wang1∗, Yuchen Liu1, Boxing Chen2, Jiajun Zhang1†, Wei Luo2, Zhongqiang Huang2, Chengqing Zong1 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2 Machine Intelligence Technology Lab, Alibaba DAMO Academy {wangchen2020,yuchen.liu}@ia.ac.cn, {jjzhang,cqzong}@nlpr.ia.ac.cn {Boxing.cbx,muzhuo.lw,z.huang}@alibaba-inc.com Abstract End-to-end Speech Translation (ST) aims at translating the source language speech into tar- get language text without generating the inter- mediate transcriptions. However, the training of end-to-end methods relies on parallel ST data, which are difficult and expensive to ob- tain. Fortunately, the supervised data for auto- matic speech recognition (ASR) and machine translation (MT) are usually more accessible, making zero-shot speech translation a poten- tial direction. Existing zero-shot methods fail to align the two modalities of speech and text into a shared semantic space, resulting in much worse performance compared to the supervised ST methods. In order to enable zero-shot ST, we propose a novel Discrete Cross-Modal Alignment (DCMA) method that employs a shared discrete vocabulary space to accommo- date and match both modalities of speech and text. Specifically, we introduce a vector quan- tization module to discretize the continuous representations of speech and text into a fi- nite set of virtual tokens, and use ASR data to map corresponding speech and text to the same virtual token in a shared codebook. This way, source language speech can be embedded in the same semantic space as the source lan- guage text, which can be then transformed into target language text with an MT module. Exper- iments on multiple language pairs demonstrate that our zero-shot ST method significantly im- proves the SOTA, and even performs on par with the strong supervised ST baselines1. 1 Introduction End-to-end Speech Translation (ST) aims at de- signing a single model to directly learn the map- ping between source language speech and target language text, and has attracted much attention ∗Work was done while at Alibaba DAMO Academy. †Corresponding author. 1Our code is available at https://github.com/ ZNLP/zero-shot-st source inputs shared semantic space translation(ich liebe dich) speech(i love you) text(i love you) shared(i love you) encoder encoder decoder speech distribution text distribution shared distribution Figure 1: The key of zero-shot speech translation is to learn an appropriate shared semantic space for source language speech and text. recently due to its advantages of no error propa- gation and lower decoding latency (Berard et al., 2016; Liu et al., 2019; Wang et al., 2020c; Xu et al., 2021). However, the training of end-to-end mod- els requires large-scale and high-quality parallel ST data, which are expensive and difficult to ob- tain. Public datasets such as MUST-C (Gangi et al., 2019a) and CoVoST (Wang et al., 2020a) are quite limited in scale and languages. In contrast, the datasets for automatic speech recognition (ASR) and machine translation (MT) are easier to access in practice. Therefore, zero-shot ST, which learns an end-to-end model …

Content truncated to 500 words