Using a new analytic measure for the annotation and analysis of MT errors on real data

Using a New Analytic Measure for the Annotation and Analysis of MT Errors on Real Data Arle Lommel1, Aljoscha Burchardt1, Maja Popovi´c1, Kim Harris2, Eleftherios Avramidis1, Hans Uszkoreit1 1DFKI / Berlin, Germany 2 text&form / Berlin, Germany name.surname@dfki.de1 kim harris@textform.com2 Abstract This work presents the new flexible Mul- tidimensional Quality Metrics (MQM) framework and uses it to analyze the per- formance of state-of-the-art machine trans- lation systems, focusing on “nearly accept- able” translated sentences. A selection of WMT news data and “customer” data provided by language service providers (LSPs) in four language pairs was anno- tated using MQM issue types and exam- ined in terms of the types of errors found in it. Despite criticisms of WMT data by the LSPs, an examination of the resulting er- rors and patterns for both types of data shows that they are strikingly consistent, with more variation between language pairs and system types than between text types. These results validate the use of WMT data in an analytic approach to as- sessing quality and show that analytic ap- proaches represent a useful addition to more traditional assessment methodolo- gies such as BLEU or METEOR. 1 Introduction For a number of years, the Machine Translation (MT) community has used “black-box” measures of translation performance like BLEU (Papineni et al., 2002) or METEOR (Denkowski and Lavie, 2011). These methods have a number of advan- tages in that they can provide automatic scores for c⃝ 2014 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. MT output in cases where there are existing refer- ence translations by calculating similarity between the MT output and the references. However, such metrics do not provide insight into the specific nature of problems encountered in the translation output and scores are tied to the particularities of the reference translations. As a result of these limitations, there has been a recent shift towards the use of more explicit er- ror classification and analysis (see, e.g., Vilar et al. (2006)) in addition to automatic metrics. The error profiles used, however are typically ad hoc cate- gorizations and specific to individual MT research projects, thus limiting their general usability for research or comparability with human translation (HT) results. In this paper, we will report on an- notation experiments that use a new, flexible er- ror metric and that showcase a new type of MT research involving collaboration between MT re- searchers, human translators, and Language Ser- vice Providers (LSPs). When we started to prepare our annota- tion experiments, we teamed up with LSPs and designed a custom error metric based on the “Multidimensional Quality Metric” MQM designed by the QTLaunchPad project (http://www.qt21.eu/launchpad). The metric was designed to facilitate annotation of MT output by human translators while containing analytic error classes we considered relevant to MT research (see Section 2, below). This paper represents the first publication of results from use of MQM for MT quality analysis. Previous research in this area has used er- …

Content truncated to 500 words