Automatic error classification with multiple error labels

Automatic error classification with multiple error labels Maja Popovi´c ADAPT Centre Dublin City University Ireland maja.popovic@adaptcentre.ie David Vilar Amazon Germany dvilar@amazon.com Abstract Although automatic classification of ma- chine translation errors still cannot provide the same detailed granularity as manual er- ror classification, it is an important task which enables estimation of translation er- rors and better understanding of the ana- lyzed MT system, in a short time and on a large scale. State-of-the-art methods use hard decisions to assign single error labels to each word. This work presents first re- sults of a new error classification method, which assigns multiple error labels to each word. We assign fractional counts for each label, which can be interpreted as a con- fidence for the label. Our method gener- ates sensible multi-error suggestions, and improves the correlation between manual and automatic error distributions. 1 Introduction Translations produced by machine transla- tion (MT) systems have been evaluated mostly in terms of overall performance scores, either by manual evaluations (ALPAC, 1966; White et al., 1994; Graham et al., 2017; Federmann, 2018) or by automatic metrics (Papineni et al., 2002; Lavie and Denkowski, 2009; Snover et al., 2006; Popovi´c, 2015; Wang et al., 2016). All these overall scores give an indication of the general performance of a given system, but they do not provide any additional information. Translation error analysis, both manual (Vilar et al., 2006; Farr´us et al., 2010; Lommel et al., c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 2014b) as well as automatic (Popovi´c and Ney, 2011; Zeman et al., 2011), as a way to identify weaknesses of the systems and define priorities for their improvement, has received a fair amount of attention in the MT community. Although automatic error classification still cannot deal with fine-grained error taxonomies, it represents a valuable tool for fast and large scale translation error analysis. With the emergence of neural MT systems, first insights about the differences between the neural approach and the then state- of-the-art statistical phrase-based approach were obtained by using automatic error classification. Bentivogli et al. (2016) analyzed four MT systems for English into German by comparing different TER (Snover et al., 2006) scores and sub-scores, and Toral and S´anchez-Cartagena (2017) applied the WER-based approach proposed by Popovi´c and Ney (2011) for a multilingual and multi- faceted evaluation of eighteen MT systems for nine translation directions including six languages from four different families. So far, automatic error classification is based on hard decisions about the error class for a given word. Addicter (Zeman et al., 2011) uses a first-order Markov model for aligning reference words with hypothesis words, and Popovi´c and Ney (2011) use WER alignments; both meth- ods assign only one single error label for each word. However, the assumption that each word can be tagged with only one error category can be somewhat restrictive. Human annotators’ feed- back (Popovi´c and Burchardt, 2011; Lommel et al., 2014a; Klubiˇcka et al., 2018) …

Content truncated to 500 words