SimulSpeech: End-to-End Simultaneous Speech to Text Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3787–3796 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3787 SimulSpeech: End-to-End Simultaneous Speech to Text Translation Yi Ren ∗ Zhejiang University rayeren@zju.edu.cn Jinglin Liu ∗ Zhejiang University jinglinliu@zju.edu.cn Xu Tan ∗ Microsoft Research xuta@microsoft.com Chen Zhang ∗ Zhejiang University zc99@zju.edu.cn Tao Qin Microsoft Research taoqin@microsoft.com Zhou Zhao† Zhejiang University zhaozhou@zju.edu.cn Tie-Yan Liu Microsoft Research tyliu@microsoft.com Abstract In this work, we develop SimulSpeech, an end- to-end simultaneous speech to text translation system which translates speech in source lan- guage to text in target language concurrently. SimulSpeech consists of a speech encoder, a speech segmenter and a text decoder, where 1) the segmenter builds upon the encoder and leverages a connectionist temporal classifica- tion (CTC) loss to split the input streaming speech in real time, 2) the encoder-decoder at- tention adopts a wait-k strategy for simulta- neous translation. SimulSpeech is more chal- lenging than previous cascaded systems (with simultaneous automatic speech recognition (ASR) and simultaneous neural machine trans- lation (NMT)). We introduce two novel knowl- edge distillation methods to ensure the perfor- mance: 1) Attention-level knowledge distilla- tion transfers the knowledge from the multipli- cation of the attention matrices of simultane- ous NMT and ASR models to help the training of the attention mechanism in SimulSpeech; 2) Data-level knowledge distillation transfers the knowledge from the full-sentence NMT model and also reduces the complexity of data distri- bution to help on the optimization of Simul- Speech. Experiments on MuST-C English- Spanish and English-German spoken language translation datasets show that SimulSpeech achieves reasonable BLEU scores and lower delay compared to full-sentence end-to-end speech to text translation (without simultane- ous translation), and better performance than the two-stage cascaded simultaneous transla- tion model in terms of BLEU scores and trans- lation delay. ∗ Equal contribution. † Corresponding author 1 Introduction Simultaneous speech to text translation (F¨ugen et al., 2007; Oda et al., 2014; Dalvi et al., 2018), which translates source-language speech into target- language text concurrently, is of great importance to the real-time understanding of spoken lectures or conversations and now widely used in many scenar- ios including live video streaming and international conferences. However, it is widely considered as one of the challenging tasks in machine transla- tion domain because simultaneous speech to text translation has to understand the speech and trade off translation accuracy and delay. Conventional approaches to simultaneous speech to text transla- tion (F¨ugen et al., 2007; Oda et al., 2014; Dalvi et al., 2018) divide the translation process into two stages: simultaneous automatic speech recognition (ASR) (Rao et al., 2017) and simultaneous neu- ral machine translation (NMT) (Gu et al., 2016), which cannot be optimized jointly and result in in- ferior accuracy, and also incurs more translation delay due to two stages. In this paper, we move a step further to translate the source speech to target text simultaneously, and develop SimulSpeech, an end-to-end simultaneous speech to text translation system. The SimulSpeech model consists of …

Content truncated to 500 words