The DCU Dependency-Based Metric in WMT-MetricsMATR 2010

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 349–353, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics The DCU Dependency-Based Metric in WMT-MetricsMATR 2010 Yifan He Jinhua Du Andy Way Josef van Genabith Centre for Next Generation Localisation School of Computing Dublin City University Dublin 9, Ireland {yhe,jdu,away,josef}@computing.dcu.ie Abstract We describe DCU’s LFG dependency- based metric submitted to the shared eval- uation task of WMT-MetricsMATR 2010. The metric is built on the LFG F-structure- based approach presented in (Owczarzak et al., 2007). We explore the following improvements on the original metric: 1) we replace the in-house LFG parser with an open source dependency parser that directly parses strings into LFG depen- dencies; 2) we add a stemming module and unigram paraphrases to strengthen the aligner; 3) we introduce a chunk penalty following the practice of METEOR to re- ward continuous matches; and 4) we intro- duce and tune parameters to maximize the correlation with human judgement. Exper- iments show that these enhancements im- prove the dependency-based metric’s cor- relation with human judgement. 1 Introduction String-based automatic evaluation metrics such as BLEU (Papineni et al., 2002) have led directly to quality improvements in machine translation (MT). These metrics provide an alternative to ex- pensive human evaluations, and enable tuning of MT systems based on automatic evaluation results. However, there is widespread recognition in the MT community that string-based metrics are not discriminative enough to reflect the translation quality of today’s MT systems, many of which have gone beyond pure string-based approaches (cf. (Callison-Burch et al., 2006)). With that in mind, a number of researchers have come up with metrics which incorporate more so- phisticated and linguistically motivated resources. Examples include METEOR (Banerjee and Lavie, 2005; Lavie and Denkowski, 2009) and TERP (Snover et al., 2010), both of which now uti- lize stemming, WordNet and paraphrase informa- tion. Experimental and evaluation campaign re- sults have shown that these metrics can obtain bet- ter correlation with human judgements than met- rics that only use surface-level information. Given that many of today’s MT systems incor- porate some kind of syntactic information, it was perhaps natural to use syntax in automatic MT evaluation as well. This direction was first ex- plored by (Liu and Gildea, 2005), who used syn- tactic structure and dependency information to go beyond the surface level matching. Owczarzak et al. (2007) extended this line of research with the use of a term-based encoding of Lexical Functional Grammar (LFG:(Kaplan and Bresnan, 1982)) labelled dependency graphs into unordered sets of dependency triples, and calculat- ing precision, recall, and F-score on the triple sets corresponding to the translation and reference sen- tences. With the addition of partial matching and n-best parses, Owczarzak et al. (2007)’s method considerably outperforms Liu and Gildea’s (2005) w.r.t. correlation with human judgement. The EDPM metric (Kahn et al., 2010) im- proves this line of research by using arc labels derived from a Probabilistic Context-Free Gram- mar (PCFG) parse to replace the LFG …

Content truncated to 500 words