CTC-based Non-autoregressive Speech Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13321–13339 July 9-14, 2023 ©2023 Association for Computational Linguistics CTC-based Non-autoregressive Speech Translation Chen Xu1†, Xiaoqian Liu1, Xiaowen Liu1, Qingxuan Sun1, Yuhao Zhang1, Murun Yang1, Qianqian Dong2, Tom Ko2, Mingxuan Wang2∗, Tong Xiao1,3∗, Anxiang Ma1,3, Jingbo Zhu1,3 1School of Computer Science and Engineering, Northeastern University, Shenyang, China 2ByteDance 3NiuTrans Research, Shenyang, China {xuchennlp, liuxiaoqian0319, liuxiaowenneu}@outlook.com {dongqianqian, tom.ko, wangmingxuan.89}@bytedance.com {xiaotong, maanxiang, zhujingbo}@mail.neu.edu.cn Abstract Combining end-to-end speech translation (ST) and non-autoregressive (NAR) generation is promising in language and speech processing for their advantages of less error propagation and low latency. In this paper, we investigate the potential of connectionist temporal classi- fication (CTC) for non-autoregressive speech translation (NAST). In particular, we develop a model consisting of two encoders that are guided by CTC to predict the source and tar- get texts, respectively. Introducing CTC into NAST on both language sides has obvious chal- lenges: 1) the conditional independent gen- eration somewhat breaks the interdependency among tokens, and 2) the monotonic alignment assumption in standard CTC does not hold in translation tasks. In response, we develop a prediction-aware encoding approach and a cross-layer attention approach to address these issues. We also use curriculum learning to im- prove convergence of training. Experiments on the MuST-C ST benchmarks show that our NAST model achieves an average BLEU score of 29.5 with a speed-up of 5.67×, which is comparable to the autoregressive counterpart and even outperforms the previous best result of 0.9 BLEU points1. 1 Introduction End-to-end speech translation (E2E ST) has at- tracted unprecedented attention and achieved dra- matic development in recent years (Duong et al., 2016; Berard et al., 2016; Weiss et al., 2017; Anas- tasopoulos and Chiang, 2018; Wang et al., 2020b,c; Xu et al., 2021; Zhang et al., 2022b). Stand-alone *Corresponding author. †Work was done while at ByteDance AI Lab. 1The code is available at https://github.com/xuchennlp/ S2T. modeling reduces the inference latency by almost half compared to cascaded systems, where the au- tomatic speech recognition (ASR) model and the machine translation (MT) model run serially. This helps the application in real scenarios, especially with limited computational resources. However, this advantage only holds in the con- text of autoregressive (AR) decoding, where each token is generated depending on the previously pre- dicted results. Non-autoregressive (NAR) genera- tion (Gu et al., 2018), the recently popular decod- ing method in ASR and MT, makes the inference process fast by predicting the output sequence in parallel, resulting in the E2E ST no longer being superior in terms of inference speed-up. A natural question arises: can we build a power- ful non-autoregressive speech translation (NAST) model? The NAR results in the latest literature are still inferior to the AR counterparts with a large gap of about 2 ∼ 3 BLEU points, even with the iterative refinement process (Inaguma et al., 2021a). In this work, we aim to develop a promising NAST model for comparable performance to the AR model with- …

Content truncated to 500 words