Curriculum Pre-training for End-to-End Speech Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3728–3738 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3728 Curriculum Pre-training for End-to-End Speech Translation Chengyi Wang∗1, Yu Wu2, Shujie Liu2, Ming Zhou2, Zhenglu Yang1 1Nankai University, Tianjin, China 2Microsoft Research Asia, Beijing, China cywang@mail.nankai.edu.cn, Wu.Yu@microsoft.com, shujliu@microsoft.com, mingzhou@microsoft.com, yangzl@nankai.edu.cn Abstract End-to-end speech translation poses a heavy burden on the encoder because it has to tran- scribe, understand, and learn cross-lingual se- mantics simultaneously. To obtain a power- ful encoder, traditional methods pre-train it on ASR data to capture speech features. How- ever, we argue that pre-training the encoder only through simple speech recognition is not enough, and high-level linguistic knowledge should be considered. Inspired by this, we pro- pose a curriculum pre-training method that in- cludes an elementary course for transcription learning and two advanced courses for under- standing the utterance and mapping words in two languages. The difficulty of these courses is gradually increasing. Experiments show that our curriculum pre-training method leads to significant improvements on En-De and En-Fr speech translation benchmarks. 1 Introduction Speech-to-Text translation (ST) is essential to breaking the language barrier for communication. It aims to translate a segment of source language speech to the target language text. To perform this task, prior works either employ a cascaded method, where an automatic speech recognition (ASR) model and a machine translation (MT) mod- el are chained together, or an end-to-end approach, where a single model converts the source language audio sequence to the target language text sequence directly (Berard et al., 2016). Due to the alleviation of error propagation and lower latency, the end-to-end ST model has been a hot topic in recent years. However, large paired data of source audios and target sentences are re- quired to train such a model, which is not easy to satisfy for most language pairs. To address this ∗Works are done during internship at Microsoft (a) previous encoder pre-training (b) curriculum encoder pre-training Figure 1: Comparison between previous encoder pre-training method with our curriculum pre-training method. issue, previous works resort to pre-training tech- nique (Berard et al., 2018; Bansal et al., 2019), where they leverage the available ASR and MT data to pre-train an ASR model and an MT mod- el respectively, and then initialize the ST model with the ASR encoder and the MT decoder. This strategy can bring faster convergence and better results. The end-to-end ST encoder has three essential roles: transcribe the speech, extract the syntactic and semantic knowledge of the source sentence and then map it to a semantic space, based on which the decoder can generate the correct target sen- tence. These pose a heavy burden to the encoder, which can be alleviated by pre-training. Howev- er, we argue that the current pre-training method restricts the power of pre-trained representations. The encoder pre-trained on the ASR task mainly 3729 focuses on transcription, which learns the align- ment between the acoustic feature with phonemes or words. It cannot capture linguistic …

Content truncated to 500 words