Context-Aware Cross-Attention for Non-Autoregressive Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4396–4402 Barcelona, Spain (Online), December 8-13, 2020 4396 Context-Aware Cross-Attention for Non-Autoregressive Translation Liang Ding†∗ Longyue Wang‡ Di Wu§ Dacheng Tao† Zhaopeng Tu‡ †The University of Sydney ldin3097@uni.sydney.edu.au dacheng.tao@sydney.edu.au ‡Tencent AI Lab §Peking University {vinnylywang,zptu}@tencent.com inbath@163.com Abstract Non-autoregressive translation (NAT) significantly accelerates the inference process by predicting the entire target sequence. However, due to the lack of target dependency modelling in the decoder, the conditional generation process heavily depends on the cross-attention. In this paper, we reveal a localness perception problem in NAT cross-attention, for which it is difficult to adequately capture source context. To alleviate this problem, we propose to enhance signals of neighbour source tokens into conventional cross-attention. Experimental results on several representative datasets show that our approach can consistently improve translation quality over strong NAT baselines. Extensive analyses demonstrate that the enhanced cross-attention achieves better exploitation of source contexts by leveraging both local and global information. 1 Introduction Different from autoregressive translation (Bahdanau et al., 2015; Vaswani et al., 2017, AT) models that generate each target word conditioned on previously generated ones, non-autoregressive translation (Gu et al., 2018, NAT) models break the autoregressive factorization and produce the target words in parallel. Given a source sentence x, the probability of generating its target sentence y with length T is defined by NAT as: p(y|x) = pL(T|x; θ) �T t=1 p(yt|x; θ), where pL(·) is a separate conditional distribution to predict the length of target sequence. As NAT models can predict all tokens independently and simultaneously, recent works have fully investigated their superiority on decoding efficiency (Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019; Kasai et al., 2020; Sun et al., 2019; Shu et al., 2020; Ran et al., 2019). However, there still exists a gap between AT and NAT models in terms of effectiveness. In encoder-decoder frameworks, the cross-attention module dynamically selects relevant source-side information (key) given a target-side token (query) (Yang et al., 2020; Wang and Tu, 2020). Through qualitative and quantitative analyses, we found that it is difficult for the NAT decoder to adequately capture the source context due to the lack of autoregressive factorization. As shown in Table 1, when translating the Chinese word “交往”, the source context word “女孩” should play a significant role in predicting the candidate word “dating”. However, the NAT model inappropriately generates “socializing with”, resulting in lexical choice errors. As seen, the AT model gives relatively higher attention weights to local contexts on the source side while the NAT model pays less attention on them (0.15 vs 0.04). We make further statistical analysis in Section 2 to prove the universality of this localness perception problem. Similar to our findings, Li et al. (2019) showed that distributions of cross-attention in NAT models are more ambiguous than those in AT ones. To alleviate this localness perception problem in NAT, we propose a context-aware cross-attention to model both local and global contexts simultaneously. For local attention, we limit the …

Content truncated to 500 words