Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2619–2630 August 1–6, 2021. ©2021 Association for Computational Linguistics 2619 Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders Chen Xu1, Bojie Hu2, Yanyang Li3, Yuhao Zhang1, Shen Huang2, Qi Ju2∗, Tong Xiao1,4∗, Jingbo Zhu1,4 1NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China 2Tencent Minority-Mandarin Translation, Beijing, China 3The Chinese University of Hong Kong, Hong Kong, China 4NiuTrans Research, Shenyang, China {xuchenneu,blamedrlee,yoohaozhang}@outlook.com, {bojiehu,springhuang,damonju}@tencent.com, {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract Encoder pre-training is promising in end-to- end Speech Translation (ST), given the fact that speech-to-translation data is scarce. But ST encoders are not simple instances of Au- tomatic Speech Recognition (ASR) or Ma- chine Translation (MT) encoders. For exam- ple, we find that ASR encoders lack the global context representation, which is necessary for translation, whereas MT encoders are not de- signed to deal with long but locally attentive acoustic sequences. In this work, we pro- pose a Stacked Acoustic-and-Textual Encoding (SATE) method for speech translation. Our en- coder begins with processing the acoustic se- quence as usual, but later behaves more like an MT encoder for a global representation of the input sequence. In this way, it is straight- forward to incorporate the pre-trained models into the system. Also, we develop an adaptor module to alleviate the representation incon- sistency between the pre-trained ASR encoder and MT encoder, and develop a multi-teacher knowledge distillation method to preserve the pre-training knowledge. Experimental results on the LibriSpeech En-Fr and MuST-C En- De ST tasks show that our method achieves state-of-the-art BLEU scores of 18.3 and 25.2. To our knowledge, we are the first to develop an end-to-end ST system that achieves compa- rable or even better BLEU performance than the cascaded ST counterpart when large-scale ASR and MT data is available1. 1 Introduction End-to-end Speech Translation (E2E ST) has be- come popular recently for its ability to free design- ers from cascading different systems and shorten ∗Corresponding author 1The source code is available at https://github.com/xuchen neu/SATE Setting Model BLEU Restricted Cascaded 23.3 E2E+Pre-training 23.1 Unrestricted Cascaded 28.1 E2E+Pre-training 25.6 Table 1: BLEU scores [%] of a cascaded ST model and an end-to-end ST model with pre-training on the MuST- C En-De corpus. Restricted = training is restricted to the ST data, and Unrestricted = additional training data is allowed for ASR and MT. the pipeline of translation (Duong et al., 2016; Be- rard et al., 2016; Weiss et al., 2017). Promising results on small-scale tasks are generally favor- able. However, speech-to-translation paired data is scarce. Researchers typically use pre-trained Au- tomatic Speech Recognition (ASR) and Machine Translation (MT) models to boost ST systems (Be- rard et al., 2018). For example, one can initialize the ST encoder using a large-scale ASR model (Bansal et al., 2019). But we note that, despite significant development effort, our end-to-end ST system with pre-trained models was not able to outperform the cascaded …

Content truncated to 500 words