Are Automatic Metrics Robust and Reliable in Specific Machine Translation Tasks?

Are Automatic Metrics Robust and Reliable in Specific Machine Translation Tasks? Mara Chinea-Rios ´Alvaro Peris Pattern Recognition and Human Language Technology Research Center Universitat Polit`ecnica de Val`encia, Val`encia, Spain {machirio, lvapeab, fcn}@prhlt.upv.es Francisco Casacuberta Abstract We present a comparison of automatic metrics against human evaluations of translation quality in several scenarios which were unexplored up to now. Our experimentation was conducted on transla- tion hypotheses that were problematic for the automatic metrics, as the results greatly diverged from one metric to another. We also compared three different translation technologies. Our evaluation shows that in most cases, the metrics capture the human criteria. However, we face failures of the automatic metrics when applied to some domains and systems. Interestingly, we find that au- tomatic metrics applied to the neural ma- chine translation hypotheses provide the most reliable results. Finally, we provide some advice when dealing with these prob- lematic domains. 1 Introduction Machine translation (MT) assessment is an open research question. The most accurate methods re- quire a manual evaluation of the MT system. Un- fortunately, this is a difficult and costly process, being unaffordable while developing new MT en- gines. Therefore, protocols for automatic evalu- ation of MT are required. The most common ap- proach for evaluating the MT quality is to compare the system hypotheses with one or more reference sentences and compute a quality score. c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. A significant research effort has been spent on enhancing the automatic metrics. For in- stance, a shared task is running since 2008, as part of the Conference on Machine Translation (WMT). Although several metrics have been pro- posed, the literature is nowadays dominated by BLEU (BiLingual Evaluation Understudy) (Pap- ineni et al., 2002) and, to a lesser extent, by TER (Translation Edit Rate) (Snover et al., 2006). Despite their usefulness, those metrics may di- verge, sometimes leading to deceiving conclu- sions. This is the case of unconventional tasks or domains (e.g Chinea-Rios et al. (2017)). This work aims to shed some light on these be- haviors, by conducting a human evaluation of MT outputs that produce inconsistencies in the met- rics. More precisely, we study the correlation be- tween human judgment and automatic evaluation on three problematic domains and for three dif- ferent MT systems. We analyze the strengths and flaws that each automatic metric conveys, giving some advice for future research. The main contri- butions of this paper are the following: • We deepen into an unexplored field: the eval- uation of MT outputs which present inconsis- tencies between automatic metrics. • We conduct a human evaluation of MT hy- pothesis which produced inconsistent auto- matic evaluations, following the direct assess- ment (DA) methodology. • We compare a large number of state-of-the- art automatic metrics for our tasks at hand. • We study the correlation of all metrics with human judgments, finding out that automatic metrics capture relatively well the human evaluation …

Content truncated to 500 words