Gaussian Multi-head Attention for Simultaneous Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 3019 - 3030 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Gaussian Multi-head Attention for Simultaneous Machine Translation Shaolei Zhang 1,2, Yang Feng 1,2∗ 1Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences, Beijing, China {zhangshaolei20z, fengyang}@ict.ac.cn Abstract Simultaneous machine translation (SiMT) out- puts translation while receiving the streaming source inputs, and hence needs a policy to deter- mine where to start translating. The alignment between target and source words often implies the most informative source word for each tar- get word, and hence provides the unified con- trol over translation quality and latency, but unfortunately the existing SiMT methods do not explicitly model the alignment to perform the control. In this paper, we propose Gaussian Multi-head Attention (GMA) to develop a new SiMT policy by modeling alignment and trans- lation in a unified manner. For SiMT policy, GMA models the aligned source position of each target word, and accordingly waits until its aligned position to start translating. To inte- grate the learning of alignment into the transla- tion model, a Gaussian distribution centered on predicted aligned position is introduced as an alignment-related prior, which cooperates with translation-related soft attention to determine the final attention. Experiments on En→Vi and De→En tasks show that our method outper- forms strong baselines on the trade-off between translation and latency. 1 Introduction Simultaneous machine translation (SiMT) (Gu et al., 2017; Ma et al., 2019; Arivazhagan et al., 2019), which outputs translation before receiving the complete source sentence, is mainly used for streaming translation tasks, such as simultaneous interpretation, live broadcast and online transla- tion. Different from full-sentence machine transla- tion which waits for the complete source sentence, SiMT requires a policy to determine where to start translating when given the streaming inputs. The SiMT policy has to trade off between translation quality and latency and an ideal one should wait ∗ Corresponding author: Yang Feng. Code is available at: https://github.com/ictnlp/ GMA Source: Target: predict action READ READ READ WRITE (a) Predict READ/WRITE action Translate Source: Target: … Wait until predict incremental step (b) Predict incremental step Figure 1: Comparison diagram of previous and pro- posed SiMT policy. The previous policies always pre- dict the READ/WRITE action step by step, while our method directly predicts incremental step (i.e., number of waiting words) between two adjacent target words. for the right number of source words, which are sufficient but not excess, until deciding to output target words (Arivazhagan et al., 2019). For full-sentence translation, each target word is generated based on the attended source informa- tion, where each source word provides different amount of information for the target word. Among them, the most informative source word can be con- sidered as an aligned word to the target word (Garg et al., 2019). Then for SiMT, the alignment can be a good guider for the policy to determine where to start translating. For high …

Content truncated to 500 words