RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2461–2474 August 1–6, 2021. ©2021 Association for Computational Linguistics 2461 RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer Xingshan Zeng, Liangyou Li, Qun Liu Huawei Noah’s Ark Lab {zeng.xingshan,liliangyou,qun.liu}@huawei.com Abstract End-to-end simultaneous speech translation (SST), which directly translates speech in one language into text in another language in real- time, is useful in many scenarios but has not been fully investigated. In this work, we pro- pose RealTranS, an end-to-end model for SST. To bridge the modality gap between speech and text, RealTranS gradually downsamples the in- put speech with interleaved convolution and unidirectional Transformer layers for acoustic modeling, and then maps speech features into text space with a weighted-shrinking operation and a semantic encoder. Besides, to improve the model performance in simultaneous sce- narios, we propose a blank penalty to enhance the shrinking quality and a Wait-K-Stride-N strategy to allow local reranking during decod- ing. Experiments on public and widely-used datasets show that RealTranS with the Wait-K- Stride-N strategy outperforms prior end-to-end models as well as cascaded models in diverse latency settings. 1 Introduction Simultaneous speech translation (SST) (F¨ugen et al., 2007; Oda et al., 2014; Ren et al., 2020) aims to translate speech in one language into text in another language concurrently. It is useful in many scenarios, like synchronous interpretation in international conferences, automatic caption for live videos, etc. However, prior studies either focus on full sentence speech translation (ST) (Berard et al., 2016; Weiss et al., 2017; Liu et al., 2019) or simultaneous text-to-text machine translation (STT) (Cho and Esipova, 2016; Gu et al., 2017; Dalvi et al., 2018) which takes a segmented output from an automatic speech recognition (ASR) sys- tem as input. Such two-stage models (i.e., cascaded models) inevitably introduce error propagation and also increase translation latency (see Figure 1). Ren et al. (2020) propose an end-to-end SST system ASR NMT My mom was the …. I’m almost the …. Soy casi el …. (a) Cascaded Model … ST Decoder ST Encoder Mi madre era la …. My mom was the …. (b) RealTranS Figure 1: An example for a cascaded model and our Re- alTranS. The ASR part in the cascaded model wrongly recognizes “My mom was” as “I’m almost”. The error is propagated to NMT and leads to a wrong translation. RealTranS avoids such errors and translates accurately. called SimulSpeech, but they ignore the modality gap between speech and text, which is important for improving translation quality (Liu et al., 2020). In this paper, we propose RealTranS model for SST. To relieve the burden of our encoder (Wang et al., 2020c; Liu et al., 2020), we decouple it into three parts: acoustic encoder, weighted-shrinking operation, and semantic encoder. We apply Conv- Transformer (Huang et al., 2020) as our acoustic encoder, which gradually downsamples the input speech and learns acoustic information with inter- leaved convolution and Transformer layers. The weighted-shrinking operation bridges the length gap between speech and text, by weighted sum- …

Content truncated to 500 words