Learning Shared Semantic Space for Speech-to-Text Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2214–2225 August 1–6, 2021. ©2021 Association for Computational Linguistics 2214 Learning Shared Semantic Space for Speech-to-Text Translation Chi Han1, Mingxuan Wang1, Heng Ji2, Lei Li1 1 ByteDance AI Lab, 2 University of Illinois at Urbana-Champaign {hanchi.me, wangmingxuan.89, lilei.02}@bytedance.com hengji@illinois.edu Abstract Having numerous potential applications and great impact, end-to-end speech translation (ST) has long been treated as an independent task, failing to fully draw strength from the rapid advances of its sibling - text machine translation (MT). With text and audio inputs represented differently, the modality gap has rendered MT data and its end-to-end models incompatible with their ST counterparts. In ob- servation of this obstacle, we propose to bridge this representation gap with Chimera. By pro- jecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Lib- rispeech, to a new state-of-the-art. Specifically, Chimera obtains 27.1 BLEU on MuST-C EN- DE, improving the SOTA by a +1.9 BLEU margin. Further experimental analyses demon- strate that the shared semantic space indeed conveys common knowledge between these two tasks and thus paves a new way for aug- menting training resources across modalities. 1 1 Introduction Speech-to-text translation (ST) takes speech input in a source language and outputs text utterance in a target language. It has many real-world applica- tions, including automatic video captioning, simul- taneous translation for international conferences, etc. Traditional ST approaches cascade automatic speech recognition (ASR) and machine translation (MT) (Sperber et al., 2017, 2019; Zhang et al., 2019; Beck et al., 2019; Cheng et al., 2019). How- ever, cascaded models often suffer from the issues of error propagation and translation latency. As a result, there have been a series of recent attempts on end-to-end speech-to-text translation (Liu et al., 1All codes, data, and resources will be made released at https://github.com/Glaciohound/Chimera-SLT. 2019, 2018; Weiss et al., 2017; B´erard et al., 2018; Duong et al., 2016; Jia et al., 2019; Dong et al., 2021b; Wang et al., 2020b). The end-to-end ap- proaches learn a single unified model, which is easier to deploy, has lower latency and could po- tentially reduce errors. However, it remains a challenge for end-to-end ST to catch up with their cascaded counterparts in performance. We argue that the root cause is the gap between the two modalities, speech and text. Although they both encode human languages, they are dissimilar in both coding attributes (pitch, volume, and intonation versus words, affixes, and punctuation) and length (thousands of time frames versus tens of words). This issue is further coupled with the relatively smaller amount of parallel data for ST than for MT. To tackle these challenges, we resort to making use of the additional available bilingual data for MT. Our hypothesis is, to better leverage MT data, an ideal model should be able to bridge the repre- sentations between speech and text. Motivated by this intuition, we propose Chimera, a text-speech shared semantic …

Content truncated to 500 words