Truly Exploring Multiple References for Machine Translation Evaluation

Truly Exploring Multiple References for Machine Translation Evaluation Ying Qin Dept. of Computer Science Beijing Foreign Studies University qinying@bfsu.edu.cn Lucia Specia Dept. of Computer Science University of Sheffield l.specia@sheffield.ac.uk Abstract Multiple references in machine transla- tion evaluation are usually under-explored: they are ignored by alignment-based met- rics and treated as bags of n-grams in string matching evaluation metrics, none of which take full advantage of the recur- ring information in these references. By exploring information on the n-gram dis- tribution and on divergences in multiple references, we propose a method of n- gram weighting and implement it to gen- erate new versions of the popular BLEU and NIST metrics. Our metrics are tested in two into-English machine translation datasets. They lead to a significant in- crease in Pearson’s correlation with human fluency judgements at system-level eval- uation. The new NIST metric also out- performs the standard NIST for document- level evaluation. 1 Introduction Quality evaluation plays a critical role in Machine Translation (MT). Since its conception, the BLEU metric (Papineni et al., 2002) has had a signifi- cant impact on MT development. Although hu- man evaluation has been used in recent evalua- tion campaigns such as WMT (Workshop on Sta- tistical MT) (Bojar et al., 2014) and other forms of reference-less metrics have been proposed (Ga- mon et al., 2005; Specia et al., 2010), the merit of language and resource-independent n-gram based metrics such as BLEU is undeniable. Despite its c⃝ 2015 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. criticisms, BLEU is thus still considered the de facto or at least a baseline metric for MT quality evaluation. Due to the cost of human translation, often only one reference translation is available at evaluation time. However, generally there are numerous valid translations for a given sentence or document. Dif- ferent references provide valid variations in lin- guistic aspects such as style, word choice and word order. Therefore, having multiple reference trans- lations is key to improve the reliability of n-gram based evaluation metrics: the more references, the more chances for n-grams correctly translated to be captured. HyTER, an n-gram matching metric based on an exponential number of reference trans- lations for a given target sentence, demonstrates the potential for better machine translation evalua- tion results from having as many references as pos- sible (Dreyer and Marcu, 2012). Nevertheless, in the more realistic case where only a few references are available, if these are simply taken as bags of n-grams, increasing the number of references will not lead to the best possible results, as pointed out by Doddington (2002). In this paper we explore how to use multiple ref- erences by means other than simply viewing them as bags of n-gram like BLEU, NIST (Dodding- ton, 2002) and other n-gram co-occurrence based metrics do. Our assumption is that each reference reflects the complete meaning of the source seg- ment. The semantic entirety of the translation will be adversely affected if …

Content truncated to 500 words