Attention as a Guide for Simultaneous Speech Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13340–13356 July 9-14, 2023 ©2023 Association for Computational Linguistics Attention as a Guide for Simultaneous Speech Translation Sara Papi32, Matteo Negri3, Marco Turchi△ 3Fondazione Bruno Kessler 2University of Trento △Independent Researcher {spapi,negri}@fbk.eu, marco.turchi@gmail.com Abstract In simultaneous speech translation (SimulST), effective policies that determine when to write partial translations are crucial to reach high out- put quality with low latency. Towards this ob- jective, we propose EDATT (Encoder-Decoder Attention), an adaptive policy that exploits the attention patterns between audio source and target textual translation to guide an offline- trained ST model during simultaneous infer- ence. EDATT exploits the attention scores mod- eling the audio-translation relation to decide whether to emit a partial hypothesis or wait for more audio input. This is done under the as- sumption that, if attention is focused towards the most recently received speech segments, the information they provide can be insufficient to generate the hypothesis (indicating that the sys- tem has to wait for additional audio input). Re- sults on en→{de, es} show that EDATT yields better results compared to the SimulST state of the art, with gains respectively up to 7 and 4 BLEU points for the two languages, and with a reduction in computational-aware latency up to 1.4s and 0.7s compared to existing SimulST policies applied to offline-trained models. 1 Introduction In simultaneous speech translation (SimulST), sys- tems have to generate translations incrementally while concurrently receiving audio input. This re- quirement poses a significant challenge since the need of generating high-quality outputs has to be balanced with the need to minimize their latency, i.e. the time elapsed (lagging) between when a word is uttered and when it is actually translated by the system. In direct SimulST systems (Bérard et al., 2016; Weiss et al., 2017),1 the balance between output 1In this paper, we focus on direct models that exhibit lower latency and better performance compared to traditional cas- cade architectures composed of separate automatic speech recognition and machine translation components (Ansari et al., 2020; Anastasopoulos et al., 2021, 2022). quality and latency is managed by a decision policy, which is the strategy for determining, at each time step, whether to emit a partial translation or to wait for additional audio input. Decision policies can be divided into two categories: fixed and adaptive. Fixed policies are usually based on simple heuris- tics (Ma et al., 2019), while adaptive policies take into account the actual input content to make the decisions (Zheng et al., 2020). Recent works (Liu et al., 2021b; Zaidi et al., 2021, 2022; Zhang and Feng, 2022) proved the superiority of adaptive poli- cies over fixed ones. However, a major limitation of these policies is that they require training ad-hoc and complex SimulST architectures, which results in high computational costs. Computational costs are also inflated by the com- mon practice of simulating the simultaneous test conditions by providing partial input during train- ing to avoid the quality …

Content truncated to 500 words