Towards Efficient Machine Translation Evaluation by Modelling Annotators

Towards Efficient Machine Translation Evaluation by Modelling Annotators Nitika Mathur Timothy Baldwin Trevor Cohn School of Computing and Information Systems The University of Melbourne Victoria 3010, Australia nmathur@student.unimelb.edu.au {tbaldwin,tcohn}@unimelb.edu.au Abstract Current machine translation evaluations use Direct Assessment, based on crowd- sourced judgements from a large pool of workers, along with quality control checks, and a robust method for combin- ing redundant judgements. In this paper we show that the quality control mecha- nism is overly conservative, increasing the time and expense of the evaluation. We propose a model that does not filter work- ers, and takes into account varying anno- tator reliabilities. Our model effectively weights each worker’s scores based on the inferred precision of the worker, and is much more reliable than the mean of ei- ther the raw or standardised scores. 1 Introduction Accurate evaluation is critical for measuring progress in machine translation (MT). Despite progress over the years, automatic metrics are still biased, and human evaluation is still a fun- damental requirement for reliable evaluation. The process of collecting human annotations is time- consuming and expensive, and the data is always noisy. The question of how to efficiently collect this data has evolved over the years, but there is still scope for improvement. Furthermore, once the data has been collected, there is no consensus on the best way to reason about translation quality. Direct Assessment (“DA”: Graham et al. (2017)) is currently accepted as the best practice for human evaluation, and is the official method at the Conference for Machine Translation (Bo- jar et al., 2017a). Every annotator scores a set of translation-pairs, which includes quality control items designed to filter out unreliable workers. However, the quality control process has low recall for good workers: as demonstrated in Sec- tion 3, about one third of good data is discarded, increasing expense. Once good workers are identi- fied, their outputs are simply averaged to produce the final ‘true’ score, despite their varying accu- racy. In this paper, we provide a detailed analysis of these shortcomings of DA and propose a Bayesian model to address these issues. Instead of stan- dardising individual worker scores, our model can automatically infer worker offsets using the raw scores of all workers as input. In addition, by learning a worker-specific precision, each worker effectively has a differing magnitude of vote in the ensemble. When evaluated on the WMT 2016 Tr-En dataset which has a high proportion of un- skilled annotators, these models are more efficient than the mean of the standardised scores. 2 Background The Conference on Machine Translation (WMT) annually collects human judgements to evaluate the MT systems and metrics submitted to the shared tasks. The evaluation methodology has evolved over the years, from 5 point adequacy and fluency rating, to relative rankings (“RR”), to DA. With RR, annotators are asked to rank translations of 5 different MT systems. In earlier years, the final score of a system was the expected number of times its translations score better than transla- tions by …

Content truncated to 500 words