Unsupervised Quality Estimation for Neural Machine Translation

Unsupervised Quality Estimation for Neural Machine Translation Marina Fomicheva,1 Shuo Sun,2 Lisa Yankovskaya,3 Fr´ed´eric Blain,1 Francisco Guzm´an,5 Mark Fishel,3 Nikolaos Aletras,1 Vishrav Chaudhary,5 Lucia Specia1,4 1University of Sheffield 2Johns Hopkins University 3University of Tartu 4Imperial College London 5Facebook AI 1{m.fomicheva,f.blain,n.aletras,l.specia}@sheffield.ac.uk 2{ssun32}@jhu.edu 3{lisa.yankovskaya,fishel}@ut.ee 5{fguzman,vishrav}@fb.com Abstract Quality Estimation (QE) is an important com- ponent in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches re- quire large amounts of expert annotated data, computation, and time for training. As an alter- native, we devise an unsupervised approach to QE where no training or access to addi- tional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be ex- tracted from the MT system as a by-product of translation. By utilizing methods for uncer- tainty quantification, we achieve very good correlation with human judgments of quality, rivaling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE. 1 Introduction With the advent of neural models, Machine Trans- lation (MT) systems have made substantial prog- ress, reportedly achieving near-human quality for high-resource language pairs (Hassan et al., 2018; Barrault et al., 2019). However, translation quality is not consistent across language pairs, domains, and datasets. This is problematic for low-resource scenarios, where there is not enough training data and translation quality significantly lags behind. Additionally, neural MT (NMT) systems can be deceptive to the end user as they can generate flu- enttranslations thatdiffer in meaning fromthe orig- inal (Bentivogli et al., 2016; Castilho et al., 2017). Thus, it is crucial to have a feedback mechanism to inform users about the trustworthiness of a given MT output. Quality estimation (QE) aims to predict the quality of the output provided by an MT system at test time when no gold-standard human translation is available. State-of-the-art (SOTA) QE models require large amounts of parallel data for pre- training and in-domain translations annotated with quality labels for training (Kim et al., 2017a; Fonseca et al., 2019). However, such large collec- tions of data are only available for a small set of languages in limited domains. Current work on QE typically treats the MT system as a black box. In this paper we propose an alternative glass-box approach to QE that allows us to address the task as an unsupervised prob- lem. We posit that encoder-decoder NMT models (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) offer a rich source of infor- mation for directly estimating translation quality: (a) the output probability distribution from the NMT system (i.e., the probabilities obtained by applying the softmax function over the entire voca- bulary of the target language);and (b) the attention mechanism used during decoding. Our assumption is that the more confident the decoder is, the …

Content truncated to 500 words