Heterogeneous Automatic MT Evaluation Through Non-Parametric Metric Combinations

Heterogeneous Automatic MT Evaluation Through Non-Parametric Metric Combinations Jes´us Gim´enez and Llu´ıs M`arquez TALP Research Center, LSI Department Universitat Polit`ecnica de Catalunya Jordi Girona Salgado 1–3, E-08034, Barcelona {jgimenez,lluism}@lsi.upc.edu Abstract Combining different metrics into a single measure of quality seems the most direct and natural way to improve over the quality of individual metrics. Recently, several ap- proaches have been suggested (Kulesza and Shieber, 2004; Liu and Gildea, 2007; Al- brecht and Hwa, 2007a). Although based on different assumptions, these approaches share the common characteristic of being parametric. Their models involve a num- ber of parameters whose weight must be adjusted. As an alternative, in this work, we study the behaviour of non-parametric schemes, in which metrics are combined without having to adjust their relative im- portance. Besides, rather than limiting to the lexical dimension, we work on a wide set of metrics operating at different linguis- tic levels (e.g., lexical, syntactic and se- mantic). Experimental results show that non-parametric methods are a valid means of putting different quality dimensions to- gether, thus tracing a possible path towards heterogeneous automatic MT evaluation. 1 Introduction Automatic evaluation metrics have notably acceler- ated the development cycle of MT systems in the last decade. There exist a large number of metrics based on different similarity criteria. By far, the most widely used metric in recent literature is BLEU (Papineni et al., 2001). Other well-known metrics are WER (Nießen et al., 2000), NIST (Doddington, 2002), GTM (Melamed et al., 2003), ROUGE (Lin and Och, 2004a), METEOR (Banerjee and Lavie, 2005), and TER (Snover et al., 2006), just to name a few. All these metrics take into account informa- tion at the lexical level1, and, therefore, their re- liability depends very strongly on the heterogene- ity/representativity of the set of reference transla- tions available (Culy and Riehemann, 2003). In order to overcome this limitation several authors have suggested taking advantage of paraphrasing support (Zhou et al., 2006; Kauchak and Barzilay, 2006; Owczarzak et al., 2006). Other authors have tried to exploit information at deeper linguistic lev- els. For instance, we may find metrics based on full constituent parsing (Liu and Gildea, 2005), and on dependency parsing (Liu and Gildea, 2005; Amig´o et al., 2006; Mehay and Brew, 2007; Owczarzak et al., 2007). We may find also metrics at the level of shallow-semantics, e.g., over semantic roles and named entities (Gim´enez and M`arquez, 2007), and at the properly semantic level, e.g., over discourse representations (Gim´enez, 2007). However, none of current metrics provides, in iso- lation, a global measure of quality. Indeed, all met- rics focus on partial aspects of quality. The main problem of relying on partial metrics is that we may obtain biased evaluations, which may lead us to de- rive inaccurate conclusions. For instance, Callison- Burch et al. (2006) and Koehn and Monz (2006) have recently reported several problematic cases re- lated to the automatic evaluation of systems ori- ented towards maximizing different quality aspects. Corroborating the findings by Culy and Riehemann (2003), they …

Content truncated to 500 words