Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation Markus Freitag George Foster David Grangier Viresh Ratnakar Qijun Tan Wolfgang Macherey Google Research {freitag, fosterg, grangier, vratnakar, qijuntan, wmach}@google.com Abstract Human evaluation of modern high-quality machine translation systems is a difficult prob- lem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions. While there has been considerable research on human evaluation, the field still lacks a commonly accepted standard procedure. As a step toward this goal, we propose an evaluation methodology grounded in explicit error analysis, based on the Multidimensional Quality Metrics (MQM) framework. We carry out the largest MQM re- search study to date, scoring the outputs of top systems from the WMT 2020 shared task in two language pairs using annotations provided by professional translators with access to full document context. We analyze the resulting data extensively, finding among other results a substantially different ranking of evaluated systems from the one established by the WMT crowd workers, exhibiting a clear preference for human over machine output. Surprisingly, we also find that automatic metrics based on pre-trained embeddings can outperform hu- man crowd workers. We make our corpus publicly available for further research. 1 Introduction Like many natural language generation tasks, ma- chine translation (MT) is difficult to evaluate because the set of correct answers for each in- put is large and usually unknown. This limits the accuracy of automatic metrics, and necessitates costly human evaluation to provide a reliable gold standard for measuring MT quality and progress. Yet even human evaluation is problematic. For instance, we often wish to decide which of two translations is better, and by how much, but what should this take into account? If one translation sounds somewhat more natural than another, but contains a slight inaccuracy, what is the best way to quantify this? To what extent will different raters agree on their assessments? The complexities of evaluating translations— both machine and human—have been extensively studied, and there are many recommended best practices. However, due to expedience, human evaluation of MT is frequently carried out on isolated sentences by inexperienced raters with the aim of assigning a single score or ranking. When MT quality is poor, this can provide a useful signal; but as quality improves, there is a risk that the signal will become lost in rater noise or bias. Recent papers have argued that poor human evaluation practices have led to mislead- ing results, including erroneous claims that MT has achieved human parity (Toral, 2020; L¨aubli et al., 2018). Our key insight in this paper is that any scor- ing or ranking of translations is implicitly based on an identification of errors and other imperfec- tions. Asking raters for a single score forces them to synthesize this complex information, and can lead to rushed judgments based on partial anal- yses. Furthermore, the implicit weights assigned by raters to different types of errors may not match their importance in the current application. An …

Content truncated to 500 words