Fully Automatic Semantic MT Evaluation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 243–252, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Fully Automatic Semantic MT Evaluation Chi-kiu LO, Anand Karthik TUMULURU and Dekai WU HKUST Human Language Technology Center Department of Computer Science and Engineering Hong Kong University of Science and Technology { jackielo,aktumuluru,dekai} @cs.ust.hk Abstract We introduce the first fully automatic, fully seman- tic frame based MT evaluation metric, MEANT, that outperforms all other commonly used auto- matic metrics in correlating with human judgment on translation adequacy. Recent work on HMEANT, which is a human metric, indicates that machine translation can be better evaluated via semantic frames than other evaluation paradigms, requiring only minimal effort from monolingual humans to an- notate and align semantic frames in the reference and machine translations. We propose a surprisingly ef- fective Occam’s razor automation of HMEANT that combines standard shallow semantic parsing with a simple maximum weighted bipartite matching al- gorithm for aligning semantic frames. The match- ing criterion is based on lexical similarity scoring of the semantic role fillers through a simple con- text vector model which can readily be trained us- ing any publicly available large monolingual cor- pus. Sentence level correlation analysis, following standard NIST MetricsMATR protocol, shows that this fully automated version of HMEANT achieves significantly higher Kendall correlation with hu- man adequacy judgments than BLEU, NIST, ME- TEOR, PER, CDER, WER, or TER. Furthermore, we demonstrate that performing the semantic frame alignment automatically actually tends to be just as good as performing it manually. Despite its high performance, fully automated MEANT is still able to preserve HMEANT’s virtues of simplicity, repre- sentational transparency, and inexpensiveness. 1 Introduction We introduce the first fully automatic semantic-frame- based MT evaluation metric capable of outperforming all other commonly used automatic metrics like BLEU, NIST, METEOR, PER, CDER, WER, and TER for eval- uating translation adequacy. This work, MEANT, can be seen as a fully automated version of HMEANT, which is a human metric, introduced by Lo and Wu (2011b). De- spite its high performance, MEANT is still able to pre- serve HMEANT’s virtues of Occam’s razor simplicity, representational transparency, and inexpensiveness. For the past decade, MT evaluation has relied heavily on inexpensive automatic metrics such as BLEU (Pap- ineni et al., 2002), NIST (Doddington, 2002), METEOR (Banerjee and Lavie, 2005), PER (Tillmann et al., 1997), CDER (Leusch et al., 2006), WER (Nießen et al., 2000), and TER (Snover et al., 2006). In large part, this is be- cause automatic metrics significantly shorten the evalua- tion cycle by providing a fast, easy and cheap quantita- tive evaluation which can be effectively incorporated into modern SMT training methods. Despite the fact that HMEANT, a human metric re- cently proposed by Lo and Wu (2011b,c,d), was shown to reflect translation adequacy more accurately than all of these automatic metrics, it is unfortunately infeasible to incorporate the HMEANT metrics directly into SMT training methods, due to the non-automatic processes of (1) semantic parsing and (2) aligning …

Content truncated to 500 words