Learning Labelled Dependencies in Machine Translation Evaluation

Proceedings of the 13th Annual Conference of the EAMT, pages 44–51, Barcelona, May 2009 Learning Labelled Dependencies in Machine Translation Evaluation Yifan He and Andy Way Centre for Next Generation Localisation School of Computing Dublin City University {yhe,away}@computing.dcu.ie Abstract Recently novel MT evaluation metrics have been presented which go beyond pure string matching, and which corre- late better than other existing metrics with human judgements. Other research in this area has presented machine learning methods which learn directly from human judgements. In this paper, we present a novel combination of dependency- and machine learning-based approaches to au- tomatic MT evaluation, and demonstrate greater correlations with human judgement than the existing state-of-the-art methods. In addition, we examine the extent to which our novel method can be generalised across different tasks and domains. 1 Introduction There is no doubt that the onset of automatic evalu- ation metrics such as BLEU (Papineni et al., 2002) has led directly to improvements in quality in ma- chine translation (MT). Prior to their introduction, most results were anecdotal, or researchers had to conduct expensive human evaluations in order to validate their work. However, seven years after their introduction, there is widespread recognition in MT that these string-based metrics are not discriminative enough to reflect the translation quality of today’s systems, many of which have gone beyond n-grams (cf. (Callison-Burch et al., 2006)). With that in mind, a number of researchers have come up with metrics which are not wholly string- based. Perhaps the best-known alternative met- ric is METEOR (Banerjee and Lavie, 2005), which c⃝ 2009 European Association for Machine Translation. while still being string-based, tries to improve on the matching schemes of BLEU by incorporating synonym matching via WordNet. Given that many of today’s MT systems incor- porate some kind of syntactic information (e.g. (Chiang, 2005)), it was perhaps natural that other researchers would seek to use syntax in automatic MT evaluation as well. The first step in this di- rection was by (Liu and Gildea, 2005), who used syntactic structure and dependency information in order to see past the surface phenomena. Two of these metrics are based on matching syntactic sub- trees between the translation and the reference, and the third is based on matching headword chains, but only for unlabelled dependencies. Since then, (Owczarzak et al., 2007a; Owczarzak et al., 2007b) have extended this line of research with the use of a term-based encoding of LFG labelled dependency graphs into unordered sets of depen- dency triples, and calculating precision, recall, and f-measure on the sets corresponding to the transla- tion and reference sentences. With the addition of partial matching and n-best parses, (Owczarzak et al., 2007a; Owczarzak et al., 2007b) considerably outperform Liu and Gildea’s (2005) highest corre- lations with human judgement. Another line of research has led to machine learning methods which learn directly from hu- man judgements (Ye et al., 2007). In this pa- per, we combine the syntax (dependency)-based and the machine learning-based approaches, and show greater correlations with human …

Content truncated to 500 words