Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7297–7306 August 1–6, 2021. ©2021 Association for Computational Linguistics 7297 Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers Benjamin Marie Atsushi Fujita Raphael Rubino National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {bmarie,atsushi.fujita,raphael.rubino}@nict.go.jp Abstract This paper presents the first large-scale meta- evaluation of machine translation (MT). We annotated MT evaluations conducted in 769 research papers published from 2010 to 2020. Our study shows that practices for automatic MT evaluation have dramatically changed dur- ing the past decade and follow concerning trends. An increasing number of MT evalua- tions exclusively rely on differences between BLEU scores to draw conclusions, without performing any kind of statistical significance testing nor human evaluation, while at least 108 metrics claiming to be better than BLEU have been proposed. MT evaluations in recent papers tend to copy and compare automatic metric scores from previous work to claim the superiority of a method or an algorithm with- out confirming neither exactly the same train- ing, validating, and testing data have been used nor the metric scores are comparable. Further- more, tools for reporting standardized metric scores are still far from being widely adopted by the MT community. After showing how the accumulation of these pitfalls leads to dubious evaluation, we propose a guideline to encour- age better automatic MT evaluation along with a simple meta-evaluation scoring method to as- sess its credibility. 1 Introduction New research publications in machine translation (MT) regularly introduce new methods and algo- rithms to improve the translation quality of MT sys- tems. In the literature, translation quality is usually evaluated with automatic metrics such as BLEU (Papineni et al., 2002) and, more rarely, by humans. To assess whether an MT system performs better than another MT system, their scores given by an automatic metric are directly compared. While such comparisons between MT systems are exhib- ited in the large majority of MT papers, there are no well-defined guideline nor clear prerequisites under which a comparison between MT systems is considered valid. Consequently, we assume that evaluation in MT is conducted with different de- grees of thoroughness across papers and that evalu- ation practices have evolved over the years. What could be considered, by the research community, as a good evaluation methodology ten years ago may not be considered good today, and vice versa. This evolution has not been studied and whether MT evaluation has become better, or worse, is de- batable. On the other hand, several requirements for MT evaluation have been well-identified. For instance, the limitations of BLEU are well-known (Callison- Burch et al., 2006; Reiter, 2018; Mathur et al., 2020) and the necessity to report automatic metric scores through standardized tools, such as Sacre- BLEU, has been recognized (Post, 2018). More- over, a trustworthy evaluation may adopt statistical significance testing (Koehn, 2004) and strong base- lines (Denkowski and …

Content truncated to 500 words