Blend: a Novel Combined MT Metric Based on Direct Assessment — CASICT-DCU submission to WMT17 Metrics Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 598–603 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Blend: a Novel Combined MT Metric Based on Direct Assessment — CASICT-DCU submission to WMT17 Metrics Task Qingsong Ma1 Yvette Graham2 Shugen Wang1 Qun Liu2,1 1Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, University of Chinese Academy of Sciences 2 ADAPT Centre, School of Computing, Dublin City University maqingsong@ict.ac.cn, graham.yvette@gmail.com wangshugen@ict.ac.cn, qun.liu@dcu.ie Abstract Existing metrics to evaluate the quality of Machine Translation hypotheses take dif- ferent perspectives into account. DPM- Fcomb, a metric combining the merits of a range of metrics, achieved the best performance for evaluation of to-English language pairs in the previous two years of WMT Metrics Shared Tasks. This year, we submit a novel combined met- ric, Blend, to WMT17 Metrics task. Com- pared to DPMFcomb, Blend includes the following adaptations: i) We use DA hu- man evaluation to guide the training pro- cess with a vast reduction in required train- ing data, while still achieving improved performance when evaluated on WMT16 to-English language pairs; ii) We carry out experiments to explore the contribution of metrics incorporated in Blend, in order to find a trade-off between performance and efficiency. 1 Introduction Automatic machine translation evaluation (AMTE) has received much attention in recent years, with the aim of providing quick and stable measurements of the performance of machine translation (MT) systems. Various metrics for AMTE have been proposed and most operate via computation of the similarity between the MT hy- pothesis and the reference translation. However, different metrics focus on different perspectives in terms of measuring similarity. For lexical based metrics, BLEU (Papineni et al., 2002) and NIST (Doddington, 2002) count n-gram co-occurrence, Meteor (Denkowski and Lavie, 2014) and GTM (Melamed et al., 2003) catch different kinds of matches, ROUGE (Lin and Och, 2004) captures common subsequences, WER (Nießen et al., 2000), PER (Tillmann et al., 1997) and TER (Snover et al., 2009) compute the post-editing distance between the hypothesis and the reference translation. Syntactic based metrics mainly use shallow syntactic structures (Chan and Ng, 2008; Zhu et al., 2010), dependency tree structures or constituent tree structures (Owczarzak et al., 2007; Liu and Gildea, 2005). Semantic measures (Lo et al., 2012) and discourse similarity based metrics (Guzm´an et al., 2014) have also been proposed. Different metrics evaluate similarity between hypotheses and reference translations from vari- ous perspectives, each of which has pros and cons. One straightforward and effective method to take advantage of the merits of existing metrics is to combine quality scores assigned by these metrics, like DPMFcomb (Yu et al., 2015a). In WMT15 and WMT16 Metrics tasks, DPM- Fcomb was the best metric on average for to- English language pairs (Stanojevi´c et al., 2015; Bojar et al., 2016). DPMFcomb incorporates lex- ical, syntactic and semantic based metrics, using ranking SVM1 to train parameters of each metric score and achieves a high correlation with human evaluation. Human evaluations in terms …

Content truncated to 500 words