Towards a Better Evaluation of Metrics for Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 928–933 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 928 Towards a Better Evaluation of Metrics for Machine Translation Peter Stanchev Weiyue Wang Hermann Ney Human Language Technology and Pattern Recognition, Computer Science Department RWTH Aachen University, 52056 Aachen, Germany <surname>@i6.informatik.rwth-aachen.de Abstract An important aspect of machine translation is its evaluation, which can be achieved through the use of a variety of metrics. To compare these metrics, the workshop on statistical ma- chine translation annually evaluates metrics based on their correlation with human judge- ment. Over the years, methods for measuring correlation with humans have changed, but lit- tle research has been performed on what the optimal methods for acquiring human scores are and how human correlation can be mea- sured. In this work, the methods for evaluating metrics at both system- and segment-level are analyzed in detail and their shortcomings are pointed out. 1 Introduction In the past, machine translation (MT) metrics have been extensively studied and evaluated, at both system- and segment-level (Bojar et al., 2016, 2017; Ma et al., 2018, 2019). When performing system- level evaluation, the average score of a MT sys- tem is taken into account. Segment-level evalu- ation uses each sentence (segment) separately to compute correlation. The results of these metric evaluations are critical to the way MT metrics are perceived. In particular the correlation with human judgment is of great importance. For this reason, an understanding for the work- ings of the evaluation method is required. Propos- als to identify relevant system-level human scores have been discussed (Koehn, 2012; Sakaguchi et al., 2014), but no comprehensive analysis on this topic has been conducted. In particular, detailed studies on the segment-level evaluation are neglected, al- though it is an integral part of the metric evaluation. Since the goal of a metric is to evaluate a trans- lation as close as possible to a human’s rating, it is important to clearly define the methods of deter- mining human score and the methods of correla- tion measurement. This work aims to present an overview of the methods used in the evaluation, an- alyze their strengths and weaknesses, and propose solutions to some of the pitfalls of the methods. 2 Human Scores To measure the correlation between the score of a metric and the score of a human, a method of deter- mining human scores is required. Thus, a person has to judge the quality of a translated sentence. This is not a simple task, as different people may have different opinions about the exact quality of the translation. Another aspect to consider is that in order to calculate correlation, the score must be quantifiable in some way. Thus, the methods used to detect human judgment must use a suf- ficient number of human judges for them to be reproducible. In the Workshop on Statistical Machine Trans- lation (WMT), three different methods are used to determine the human score: direct assessment (DA) (Graham et al., 2017), …

Content truncated to 500 words