Accurate Evaluation of Segment-level Machine Translation Metrics

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1183–1191, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Accurate Evaluation of Segment-level Machine Translation Metrics Yvette Graham†‡ Nitika Mathur† Timothy Baldwin† †Department of Computing and Information Systems, The University of Melbourne ‡ADAPT Research Centre, Trinity College Dublin ygraham@scss.tcd.ie, nmathur@student.unimelb.edu.au, tb@ldwin.net Abstract Evaluation of segment-level machine transla- tion metrics is currently hampered by: (1) low inter-annotator agreement levels in human as- sessments; (2) lack of an effective mechanism for evaluation of translations of equal quality; and (3) lack of methods of significance testing improvements over a baseline. In this paper, we provide solutions to each of these chal- lenges and outline a new human evaluation methodology aimed specifically at assessment of segment-level metrics. We replicate the hu- man evaluation component of WMT-13 and reveal that the current state-of-the-art perfor- mance of segment-level metrics is better than previously believed. Three segment-level met- rics — METEOR, NLEPOR and SENTBLEU- MOSES — are found to correlate with human assessment at a level not significantly outper- formed by any other metric in both the individ- ual language pair assessment for Spanish-to- English and the aggregated set of 9 language pairs. 1 Introduction Automatic segment-level machine translation (MT) metrics have the potential to greatly advance MT by providing more fine-grained error analysis, increas- ing efficiency of system tuning methods and leverag- ing techniques for system hybridization. However, a major obstacle currently hindering the development of segment-level metrics is their evaluation. Human assessment is the gold standard against which met- rics must be evaluated, but when it comes to the task of evaluating translation quality, human annotators are notoriously inconsistent. For example, the main venue for evaluation of metrics, the annual Work- shop on Statistical Machine Translation (WMT), reports disturbingly low inter-annotator agreement levels and highlights the need for better human assessment of MT. WMT-13, for example, report Kappa coefficients ranging from 0.075 to 0.324 for assessors from crowd-sourcing services, only in- creasing to between 0.315 and 0.457 for MT re- searchers (Bojar et al., 2013a). For evaluation of metrics that operate at the system or document-level such as BLEU, inconsistency in individual human judgments can, to some degree, be overcome by ag- gregation of individual human assessments over the segments within a document. However, for evalua- tion of segment-level metrics, there is no escaping the need to boost the consistency of human annota- tion of individual segments. This motivates our analysis of current methods of human evaluation of segment-level metrics, and pro- posal of an alternative annotation mechanism. We examine the accuracy of segment scores collected with our proposed method by replicating compo- nents of the WMT-13 human evaluation (Bojar et al., 2013b), with the sole aim of optimizing agree- ment in segment scores to provide an effective gold standard for evaluating segment-level metrics. Our method also supports the use of significance test- ing of segment-level metrics, and tests applied to the WMT-13 metrics …

Content truncated to 500 words