MEANT at WMT 2013: A Tunable, Accurate yet Inexpensive Semantic Frame Based MT Evaluation Metric

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 422–428, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics MEANT at WMT 2013: A tunable, accurate yet inexpensive semantic frame based MT evaluation metric Chi-kiu LO and Dekai WU HKUST Human Language Technology Center Department of Computer Science and Engineering Hong Kong University of Science and Technology {jackielo|dekai}@cs.ust.hk Abstract The linguistically transparent MEANT and UMEANT metrics are tunable, simple yet highly effective, fully automatic ap- proximation to the human HMEANT MT evaluation metric which measures seman- tic frame similarity between MT output and reference translations. In this pa- per, we describe HKUST’s submission to the WMT 2013 metrics evaluation task, MEANT and UMEANT. MEANT is optimized by tuning a small number of weights—one for each semantic role label—so as to maximize correlation with human adequacy judgment on a devel- opment set. UMEANT is an unsuper- vised version where weights for each se- mantic role label are estimated via an in- expensive unsupervised approach, as op- posed to MEANT’s supervised method re- lying on more expensive grid search. In this paper, we present a battery of exper- iments for optimizing MEANT on differ- ent development sets to determine the set of weights that maximize MEANT’s accu- racy and stability. Evaluated on test sets from the WMT 2012/2011 metrics evalua- tion, both MEANT and UMEANT achieve competitive correlations with human judg- ments using nothing more than a monolin- gual corpus and an automatic shallow se- mantic parser. 1 Introduction We evaluate in the context of WMT 2013 the MEANT (Lo et al., 2012) and UMEANT (Lo and Wu, 2012) semantic machine translation (MT) evaluation metrics—tunable, simple yet highly ef- fective, fully-automatic semantic frame based ob- jective functions that score the degree of similarity between the MT output and the reference transla- tions via semantic role labels (SRL). Recent stud- ies (Lo et al., 2013; Lo and Wu, 2013) show that tuning MT systems against MEANT more robustly improves translation adequacy, compared to tun- ing against BLEU or TER. In the past decade, the progress of machine translation (MT) research is predominantly driven by the fast and cheap n-gram based MT eval- uation metrics, such as BLEU (Papineni et al., 2002), which assume that a good translation is one that shares the same lexical choices as the ref- erence translation. Despite enforcing fluency, it has been established that these metrics do not en- force translation utility adequately and often fail to preserve meaning closely (Callison-Burch et al., 2006; Koehn and Monz, 2006). Unlike BLEU, or other n-gram based MT evaluation metrics, MEANT adopts at outset the principle that a good translation is one from which the human readers may successfully understand at least the central meaning of the input sentence as captured by the basic event structure—“who did what to whom, when, where and why”(Pradhan et al., 2004). Lo et al. (2012) show that MEANT correlates better with human adequacy judgment than other commonly used automatic MT evaluation metrics, such as …

Content truncated to 500 words