Quality In, Quality Out: Learning from Actual Mistakes

Quality In, Quality Out: Learning from Actual Mistakes Fr´ed´eric Blain1 Nikolaos Aletras1 Lucia Specia1,2 1Department of Computer Science, University of Sheffield 2Department of Computing, Imperial College London United Kingdom {f.blain,n.aletras,l.specia}@sheffield.ac.uk Abstract Approaches to Quality Estimation (QE) of machine translation have shown promis- ing results at predicting quality scores for translated sentences. However, QE models are often trained on noisy approximations of quality annotations derived from the proportion of post-edited words in trans- lated sentences instead of direct human an- notations of translation errors. The latter is a more reliable ground-truth but more ex- pensive to obtain. In this paper, we present the first attempt to model the task of pre- dicting the proportion of actual transla- tion errors in a sentence while minimis- ing the need for direct human annotation. For that purpose, we use transfer-learning to leverage large scale noisy annotations and small sets of high-quality human an- notated translation errors to train QE mod- els. Experiments on four language pairs and translations obtained by statistical and neural models show consistent gains over strong baselines. 1 Introduction Quality Estimation (QE) for Machine Translation (MT) is the task of predicting the overall quality of an automatically generated translation e.g., on ei- ther word, sentence or document level (Blatz et al., 2004; Ueffing and Ney, 2007). In opposition to au- tomatic metrics and manual evaluation which rely on gold standard reference translations, QE mod- els can produce quality estimates on unseen data, c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. and at runtime. QE has already proven its useful- ness in many applications such as improving pro- ductivity in post-editing of MT, and recent neural- based approaches to QE have been shown to pro- vide promising performance in predicting quality of neural MT output (Fonseca et al., 2019). QE models are trained under full supervision, which requires to have quality-labelled training data at hand. Obtaining annotated data for all the domains and languages of interest is costly and of- ten impractical. As a result, QE models can suf- fer from the same limitations as neural MT mod- els themselves, such as drastic degradation of their performance on out-of-domain data. As an alter- native, QE models are often trained under weak supervision, using training instances labelled from noisy or limited sources (e.g. data labelled with automatic metrics for MT). Here, we focus on sentence-level QE, where given a pair of sentences (the source and its transla- tion), the aim is to train supervised Machine Learn- ing (ML) models that can predict a quality label as a numerical value. The most widely used label for sentence-level QE is the Human-mediated Transla- tion Edit Rate (HTER) (Snover et al., 2006), which represents the post-editing effort. HTER consists of the minimum number of edits a human language expert is required to make in order to fix the trans- lation errors in a sentence, taking values between 0 and 1. The main limitation of …

Content truncated to 500 words