Direct Exploitation of Attention Weights for Translation Quality Estimation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 955–960 November 10–11, 2021. ©2021 Association for Computational Linguistics 955 Direct Exploitation of Attention Weights for Translation Quality Estimation Lisa Yankovskaya and Mark Fishel Institute of Computer Science University of Tartu, Estonia {lisa_y,mark}@tartunlp.ai Abstract The paper presents our submission to the WMT2021 Shared Task on Quality Estimation (QE)1. We participate in sentence-level predic- tions of human judgments (Task 1) and post- editing effort (Task 2). We propose a glass- box approach based on attention weights ex- tracted from machine translation systems. In contrast to the previous works, we directly ex- plore attention weight matrices without replac- ing them with general metrics (like entropy). We show that some of our models can be trained with a small amount of a high-cost la- belled data. In the absence of training data our approach still demonstrates a moderate linear correlation, when trained with synthetic data. 1 Introduction Quality Estimation (QE, Blatz et al., 2004; Spe- cia et al., 2009) is an essential part of the machine translation (MT) pipeline, which estimates the qual- ity of the translation output without relying on any reference. Unlike the previous year, three QE sentence- level tasks were presented in the WMT2021 Shared Task (Specia et al., 2021). The goal of Task 1 is to predict direct assessments (DA), i.e. human judgments of translation quality (Graham et al., 2015), whereas in Task 2, the task is to estimate the post-editing effort required to obtain a correct translation which is measured by the HTER metric (Snover et al., 2006). The goal of Task 3 is to determine if the translation output contains at least one critical error. We propose a lightweight glass-box approach that can be applied to Task 1 and Task 2. The approach is based on using the encoder-decoder attention weight matrices as input features for su- pervised translation quality estimation. Next we 1http://www.statmt.org/wmt21/ quality-estimation-task.html describe our approach (Section 2) and evaluate it experimentally (Sections 3–5). 2 Approach There are several QE models based on atten- tion weights of neural MT systems described ear- lier (Yankovskaya et al., 2018; Fomicheva et al., 2020a,b). Their main idea is to compute the entropy of encoder-decoder attention weights for each tar- get token and then average these entropies to get a sentence-level metric: Entropy = −1 I I � i=1 J � j=1 αji log αji, where α represents attention weights, I is the num- ber of target tokens and J is the number of source tokens. Yankovskaya et al. (2018) work with attention weights extracted from LSTM (Hochreiter and Schmidhuber, 1997) MT systems. As LSTM has only one attention matrix, the approach of comput- ing entropies is straightforward. However, neural MT models based on Transformer (Vaswani et al., 2017) have several layers and heads, so the number of computed entropies equals [Layers × Heads] for each sentence, which introduces some difficulties in this approach. To overcome it, Fomicheva et al. (2020a) summarise entropies by taking the average or minimum …

Content truncated to 500 words