The Feasibility of HMEANT as a Human MT Evaluation Metric

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 52–61, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics The Feasibility of HMEANT as a Human MT Evaluation Metric Alexandra Birch Barry Haddow Ulrich Germann a.birch@ed.ac.uk bhaddow@inf.ed.ac.uk ugermann@inf.ed.ac.uk Maria Nadejde Christian Buck Philipp Koehn maria.nadejde@gmail.com cbuck@lantis.de pkoehn@inf.ed.ac.uk University of Edinburgh 10 Crichton Street Edinburgh, EH8 9AB, UK Abstract There has been a recent surge of interest in semantic machine translation, which stan- dard automatic metrics struggle to evalu- ate. A family of measures called MEANT has been proposed which uses semantic role labels (SRL) to overcome this prob- lem. The human variant, HMEANT, has largely been evaluated using correlation with human contrastive evaluations, the standard human evaluation metric for the WMT shared tasks. In this paper we claim that for a human metric to be useful, it needs to be evaluated on intrinsic proper- ties. It needs to be reliable; it needs to work across different language pairs; and it needs to be lightweight. Most impor- tantly, however, a human metric must be discerning. We conclude that HMEANT is a step in the right direction, but has some serious flaws. The reliance on verbs as heads of frames, and the assumption that annotators need minimal guidelines are particularly problematic. 1 Introduction Human evaluation is essential in machine transla- tion (MT) research because it is the ultimate way to judge system quality. Furthermore, human eval- uation is used to evaluate automatic metrics which are necessary for tuning system parameters. Un- fortunately, there is no clear consensus on which evaluation strategy is best. Humans have been asked to judge if translations are correct, to grade them and to rank them. But it is often very difficult to decide how good a translation is, when there are so many possible ways of translating a sentence. Another problem is that different types of evalua- tion might be useful for different purposes. If the MT is going to be the basis of a human transla- tor’s work-flow, then post-editing effort seems like a natural fit. However, for people using MT for gisting, what we really want is some measure of how much meaning has been retained. We clearly need a metric which tries to answer the question, how much of the meaning does the translation capture. In this paper, we explore the use of human evaluation metrics which attempt to capture the extent of this meaning retention. In particular, we consider HMEANT (Lo and Wu, 2011a), a metric that uses semantic role labels to measure how much of the “who, why, when, where” has been preserved. For HMEANT evalua- tion, annotators are instructed to identify verbs as heads of semantic frames. Then they attach role fillers to the heads and finally they align heads and role fillers in the candidate translation with those in a reference translation. In a series of pa- pers, Lo and Wu (2010, 2011b,a, 2012) explored a number of questions, evaluating HMEANT by us- ing correlation statistics to compare …

Content truncated to 500 words