HUME: Human UCCA-Based Evaluation of Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1264–1274, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics HUME: Human UCCA-Based Evaluation of Machine Translation Alexandra Birch1∗, Omri Abend2*, Ondˇrej Bojar3*, Barry Haddow1* 1School of Informatics, University of Edinburgh 2School of Computer Science and Engineering, Hebrew University of Jerusalem 3 Charles University in Prague, Faculty of Mathematics and Physics a.birch@ed.ac.uk, oabend@cs.huji.ac.il bojar@ufal.mff.cuni.cz, bhaddow@inf.ed.ac.uk Abstract Human evaluation of machine translation nor- mally uses sentence-level measures such as relative ranking or adequacy scales. However, these provide no insight into possible errors, and do not scale well with sentence length. We argue for a semantics-based evaluation, which captures what meaning components are retained in the MT output, thus providing a more fine-grained analysis of translation qual- ity, and enabling the construction and tuning of semantics-based MT. We present a novel human semantic evaluation measure, Human UCCA-based MT Evaluation (HUME), build- ing on the UCCA semantic representation scheme. HUME covers a wider range of se- mantic phenomena than previous methods and does not rely on semantic annotation of the potentially garbled MT output. We experi- ment with four language pairs, demonstrating HUME’s broad applicability, and report good inter-annotator agreement rates and correla- tion with human adequacy scores. 1 Introduction Human judgement should be the ultimate test of the quality of an MT system. Nevertheless, common measures for human MT evaluation, such as ade- quacy and fluency judgements or the relative rank- ing of possible translations, are problematic in two ways. First, as the quality of translation is multi- faceted, it is difficult to quantify the quality of the entire sentence in a single number. This is indeed ∗ All authors contributed equally to this work. reflected in the diminishing inter-annotator agree- ment (IAA) rates of human ranking measures with the sentence length (Bojar et al., 2011). Second, a sentence-level quality score does not indicate what parts of the sentence are badly translated, and so cannot inform developers in repairing these errors. These problems are partially addressed by mea- sures that decompose over parts of the evaluated translation, often words or n-grams (see §2 for a brief survey of previous work). A promising line of research decomposes metrics over semantically defined units, quantifying the similarity of the out- put and the reference in terms of their verb argu- ment structure; the most notable of these measures is HMEANT (Lo and Wu, 2011). We propose the HUME metric, a human evalua- tion measure that decomposes over UCCA semantic units. UCCA (Abend and Rappoport, 2013) is an appealing candidate for semantic analysis, due to its cross-linguistic applicability, support for rapid anno- tation, and coverage of many fundamental semantic phenomena, such as verbal, nominal and adjectival argument structures and their inter-relations. HUME operates by aggregating human assess- ments of the translation quality of individual seman- tic units in the source sentence. We are thus avoiding the semantic annotation of machine-generated text, which is often garbled or semantically unclear. This also allows the …

Content truncated to 500 words