MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 220–229, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via semantic frames Chi-kiu Lo and Dekai Wu HKUST Human Language Technology Center Department of Computer Science and Engineering Hong Kong University of Science and Technology {jackielo,dekai}@cs.ust.hk Abstract We introduce a novel semi-automated metric, MEANT, that assesses translation utility by match- ing semantic role fillers, producing scores that cor- relate with human judgment as well as HTER but at much lower labor cost. As machine transla- tion systems improve in lexical choice and flu- ency, the shortcomings of widespread n-gram based, fluency-oriented MT evaluation metrics such as BLEU, which fail to properly evaluate adequacy, become more apparent. But more accurate, non- automatic adequacy-oriented MT evaluation metrics like HTER are highly labor-intensive, which bottle- necks the evaluation cycle. We first show that when using untrained monolingual readers to annotate se- mantic roles in MT output, the non-automatic ver- sion of the metric HMEANT achieves a 0.43 corre- lation coefficient with human adequacy judgments at the sentence level, far superior to BLEU at only 0.20, and equal to the far more expensive HTER. We then replace the human semantic role annotators with au- tomatic shallow semantic parsing to further automate the evaluation metric, and show that even the semi- automated evaluation metric achieves a 0.34 corre- lation coefficient with human adequacy judgment, which is still about 80% as closely correlated as HTER despite an even lower labor cost for the evalu- ation procedure. The results show that our proposed metric is significantly better correlated with human judgment on adequacy than current widespread au- tomatic evaluation metrics, while being much more cost effective than HTER. 1 Introduction In this paper we show that evaluating machine trans- lation by assessing the translation accuracy of each argu- ment in the semantic role framework correlates with hu- man judgment on translation adequacy as well as HTER, at a significantly lower labor cost. The correlation of this new metric, MEANT, with human judgment is far supe- rior to BLEU and other automatic n-gram based evalua- tion metrics. We argue that BLEU (Papineni et al., 2002) and other automatic n-gram based MT evaluation metrics do not ad- equately capture the similarity in meaning between the machine translation and the reference translation—which, ultimately, is essential for MT output to be useful. N- gram based metrics assume that “good” translations tend to share the same lexical choices as the reference trans- lations. While BLEU score performs well in captur- ing the translation fluency, Callison-Burch et al. (2006) and Koehn and Monz (2006) report cases where BLEU strongly disagree with human judgment on translation quality. The underlying reason is that lexical similarity does not adequately reflect the similarity in meaning. As MT systems improve, the shortcomings of the n-gram based evaluation metrics are becoming more apparent. State-of-the-art MT systems are often able to output flu- ent translations …

Content truncated to 500 words