The role of artificially generated negative data for quality estimation of machine translation

The role of artificially generated negative data for quality estimation of machine translation Varvara Logacheva University of Sheffield Sheffield, United Kingdom v.logacheva@sheffield.ac.uk Lucia Specia University of Sheffield Sheffield, United Kingdom l.specia@sheffield.ac.uk Abstract The modelling of natural language tasks using data-driven methods is often hin- dered by the problem of insufficient nat- urally occurring examples of certain lin- guistic constructs. The task we address in this paper – quality estimation (QE) of machine translation – suffers from lack of negative examples at training time, i.e., examples of low quality translation. We propose various ways to artificially gener- ate examples of translations containing er- rors and evaluate the influence of these ex- amples on the performance of QE models both at sentence and word levels. 1 Introduction The task of classifying texts as “correct” or “incor- rect” often faces the problem of unbalanced train- ing sets: examples of the “incorrect” class can be very limited or even absent. In many cases, natu- rally occurring instances of these examples are rare (e.g. incoherent sentences, errors in human texts). In others, the labelling of data is a non-trivial task which requires expert knowledge. Consider the task of quality estimation (QE) of machine translation (MT) systems output. When performing binary classification of automatically translated sentences one should provide examples of both bad and good quality sentences. Good quality sentences can be taken from any parallel corpus of human translations, whereas there are very few corpora of sentences annotated as having low quality. These corpora need to be created by c⃝ 2015 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. human translators, who post-edit automatic trans- lations, mark errors in translations, or rate transla- tions for quality. This process is slow and expen- sive. It is therefore desirable to devise automatic procedures to generate negative training data for QE model learning. Previous work has followed the hypothesis that machine translations can be assumed to have low quality (Gamon et al., 2005). However, this is not the case nowadays: many translations can be con- sidered flawless. Particularly for word-level QE, it is unrealistic to presume that every single word in the MT output is incorrect. Another possibility is to use automatic quality evaluation metrics based on reference translations to provide a quality score for MT data. Metrics such as BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and ME- TEOR (Banerjee and Lavie, 2005) can be used to compare the automatic and reference translations. However, these scores can be very unreliable, es- pecially for word-level QE, as every word that dif- fers in form or position would be annotated as bad. Previous efforts have been made for negative data generation, including random generation of sentences from word distributions and the use of translations in low-ranked positions in n-best lists produced by statistical MT (SMT) systems. These methods are however unsuitable for QE at the word level, as they provide no information about the quality of …

Content truncated to 500 words