Streaming Models for Joint Speech Recognition and Translation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2533–2539 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2533 Streaming Models for Joint Speech Recognition and Translation Orion Weller1*, Matthias Sperber2, Christian Gollan2, Joris Kluivers2 1Brigham Young University 2Apple oweller@byu.edu,{m_sperber,cgollan,jkluivers}@apple.com Abstract Using end-to-end models for speech transla- tion (ST) has increasingly been the focus of the ST community. These models condense the previously cascaded systems by directly converting sound waves into translated text. However, cascaded models have the advantage of including automatic speech recognition out- put, useful for a variety of practical ST sys- tems that often display transcripts to the user alongside the translations. To bridge this gap, recent work has shown initial progress into the feasibility for end-to-end models to pro- duce both of these outputs. However, all pre- vious work has only looked at this problem from the consecutive perspective, leaving un- certainty on whether these approaches are ef- fective in the more challenging streaming set- ting. We develop an end-to-end streaming ST model based on a re-translation approach and compare against standard cascading ap- proaches. We also introduce a novel inference method for the joint case, interleaving both transcript and translation in generation and re- moving the need to use separate decoders. Our evaluation across a range of metrics capturing accuracy, latency, and consistency shows that our end-to-end models are statistically simi- lar to cascading models, while having half the number of parameters. We also find that both systems provide strong translation quality at low latency, keeping 99% of consecutive qual- ity at a lag of just under a second. 1 Introduction Speech translation (ST) is the process of translat- ing acoustic sound waves into text in a different language than was originally spoken in. This paper focuses on ST in a particular setting, as described by two characteristics: (1) We desire models that translate in a streaming fashion, where *Work done during an internship with Apple users desire the translation before the speaker has finished. This setting poses additional difficulties compared to consecutive translation, forcing sys- tems to translate without knowing what the speaker will say in the future. (2) Furthermore, the speaker may want to verify that their speech is being pro- cessed correctly, intuitively seeing a streaming tran- script while they speak (F¨ugen, 2008; Hsiao et al., 2006). For this reason, we consider models that produce both transcripts and translation jointly.1 Previous approaches to streaming ST have typi- cally utilized a cascaded system that pipelines the output of an automatic speech recognition (ASR) system through a machine translation (MT) model for the final result. These systems have been the preeminent strategy, taking the top place in recent streaming ST competitions (Pham et al., 2019; Jan et al., 2019; Elbayad et al., 2020; Ansari et al., 2020). Despite the strong performance of these cas- caded systems, there are also some problems: error propagation from ASR output to MT input (Ruiz and Federico, 2014); ASR/MT training …

Content truncated to 500 words