Attention Calibration for Transformer in Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1288–1298 August 1–6, 2021. ©2021 Association for Computational Linguistics 1288 Attention Calibration for Transformer in Neural Machine Translation Yu Lu1,2∗, Jiali Zeng3, Jiajun Zhang1,2†, Shuangzhi Wu3 and Mu Li3 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 3 Tencent Cloud Xiaowei, Beijing, China {yu.lu, jjzhang}@nlpr.ia.ac.cn {lemonzeng, frostwu, ethanlli}@tencent.com Abstract Attention mechanisms have achieved substan- tial improvements in neural machine transla- tion by dynamically selecting relevant inputs for different predictions. However, recent stud- ies have questioned the attention mechanisms’ capability for discovering decisive inputs. In this paper, we propose to calibrate the attention weights by introducing a mask perturbation model that automatically evaluates each in- put’s contribution to the model outputs. We in- crease the attention weights assigned to the in- dispensable tokens, whose removal leads to a dramatic performance decrease. The extensive experiments on the Transformer-based transla- tion have demonstrated the effectiveness of our model. We further find that the calibrated at- tention weights are more uniform at lower lay- ers to collect multiple information while more concentrated on the specific inputs at higher layers. Detailed analyses also show a great need for calibration in the attention weights with high entropy where the model is uncon- fident about its decision1. 1 Introduction Attention mechanisms have been ubiquitous in neu- ral machine translation (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017). It dynamically en- codes source-side information by inducing a con- ditional distribution over inputs, where the ones that are most relevant to the current translation are expected to receive more attention. However, many studies doubt whether highly- attended inputs have a large impact on the model outputs. On the one hand, erasing the representa- tions accorded high attention weights do not neces- sarily lead to a performance decrease (Serrano and ∗Work done while the author was an intern at Tencent. †Corresponding author. 1https://github.com/yulu-dada/Attention-calibration- NMT 远 郊 连 日 大 雪 多 人 死 亡 交 通 中 断 <EOS> 远郊 连 日 大雪 多 人 死亡 交通 中断 heavy snow in countryside caused many deaths Src: Base: deaths _ [before] deaths _ [after] deaths _ [mask] in _ [before] in _ [after] in _ [mask] heavy snow in countryside has caused many deaths and traffic interruption Ours: days of heavy snow in countryside left many deaths and transportation disrupted Ref: 远 郊 连 日 大 雪 多 人 死 亡 交 通 中 断 <EOS> Figure 1: Examples of the attention weights before and after calibration. “in ” denotes the timestep after the prediction “in”. The dashed boxes indicate the inputs which should receive more attention measured by our mask perturbation model. Smith, 2019), which can be attributed to that unim- portant words (e.g., punctuations) are frequently assigned with high attention weights (Mohanku- mar et al., 2020). On the …

Content truncated to 500 words