SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 582–587 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 582 SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation Xutai Ma Johns Hopkins University xutai ma@jhu.edu Juan Pino Facebook AI juancarabina@fb.com Philipp Koehn Johns Hopkins University phi@jhu.edu Abstract Simultaneous text translation and end-to-end speech translation have recently made great progress but little work has combined these tasks together. We investigate how to adapt simultaneous text translation methods such as wait-k and monotonic multihead attention to end-to-end simultaneous speech translation by introducing a pre-decision module. A detailed analysis is provided on the latency-quality trade-offs of combining fixed and flexible pre- decision with fixed and flexible policies. We also design a novel computation-aware latency metric, adapted from Average Lagging. 1 1 Introduction Simultaneous speech translation (SimulST) gener- ates a translation from an input speech utterance before the end of the utterance has been heard. SimulST systems aim at generating translations with maximum quality and minimum latency, tar- geting applications such as video caption transla- tions and real-time language interpreter. While great progress has recently been achieved on both end-to-end speech translation (Ansari et al., 2020) and simultaneous text translation (SimulMT) (Gris- som II et al., 2014; Gu et al., 2017; Luo et al., 2017; Lawson et al., 2018; Alinejad et al., 2018; Zheng et al., 2019b,a; Ma et al., 2020; Arivazhagan et al., 2019, 2020), little work has combined the two tasks together (Ren et al., 2020). End-to-end SimulST models feature a smaller model size, greater inference speed and fewer com- pounding errors compared to their cascade counter- part, which perform streaming speech recognition followed by simultaneous machine translation. In addition, it has been demonstrated that end-to-end SimulST systems can have lower latency than cas- cade systems (Ren et al., 2020). 1The code is available at https://github.com/ pytorch/fairseq In this paper, we study how to adapt methods developed for SimulMT to end-to-end SimulST. To this end, we introduce the concept of pre-decision module. Such module guides how to group en- coder states into meaningful units prior to making a READ/WRITE decision. A detailed analysis of the latency-quality trade-offs when combining a fixed or flexible pre-decision module with a fixed or flexible policy is provided. We also introduce a novel computation-aware latency metric, adapted from Average Lagging (AL) (Ma et al., 2019). 2 Task formalization A SimulST model takes as input a sequence of acoustic features X = [x1, ...x|X|] extracted from speech samples every Ts ms, and generates a se- quence of text tokens Y = [y1, ..., y|Y |] in a target language. Additionally, it is able to generate yi with only partial input X1:n(yi) = [x1, ...xn(yi)], where n(yi) ≤ |X| is the number of frames needed to generate the i-th target token yi. Note that n is a monotonic function, i.e. n(yi−1) ≤ n(yi). A SimulST model is …

Content truncated to 500 words