Linguistic Features for Automatic Evaluation of Heterogenous MT Systems

Proceedings of the Second Workshop on Statistical Machine Translation, pages 256–264, Prague, June 2007. c⃝2007 Association for Computational Linguistics Linguistic Features for Automatic Evaluation of Heterogenous MT Systems Jes´us Gim´enez and Llu´ıs M`arquez TALP Research Center, LSI Department Universitat Polit`ecnica de Catalunya Jordi Girona Salgado 1–3, E-08034, Barcelona {jgimenez,lluism}@lsi.upc.edu Abstract Evaluation results recently reported by Callison-Burch et al. (2006) and Koehn and Monz (2006), revealed that, in certain cases, the BLEU metric may not be a reliable MT quality indicator. This happens, for in- stance, when the systems under evaluation are based on different paradigms, and there- fore, do not share the same lexicon. The reason is that, while MT quality aspects are diverse, BLEU limits its scope to the lex- ical dimension. In this work, we suggest using metrics which take into account lin- guistic features at more abstract levels. We provide experimental results showing that metrics based on deeper linguistic informa- tion (syntactic/shallow-semantic) are able to produce more reliable system rankings than metrics based on lexical matching alone, specially when the systems under evaluation are of a different nature. 1 Introduction Most metrics used in the context of Automatic Ma- chine Translation (MT) Evaluation are based on the assumption that ‘acceptable’ translations tend to share the lexicon (i.e., word forms) in a predefined set of manual reference translations. This assump- tion works well in many cases. However, several results in recent MT evaluation campaigns have cast some doubts on its general validity. For instance, Callison-Burch et al. (2006) and Koehn and Monz (2006) reported and analyzed several cases of strong disagreement between system rankings provided by human assessors and those produced by the BLEU metric (Papineni et al., 2001). In particular, they noted that when the systems under evaluation are of a different nature (e.g., rule-based vs. statistical, human-aided vs. fully automatical, etc.) BLEU may not be a reliable MT quality indicator. The reason is that BLEU favours MT systems which share the ex- pected reference lexicon (e.g., statistical systems), and penalizes those which use a different one. Indeed, the underlying cause is much simpler. In general, lexical similarity is nor a sufficient neither a necessary condition so that two sentences convey the same meaning. On the contrary, natural lan- guages are expressive and ambiguous at different levels. Consequently, the similarity between two sentences may involve different dimensions. In this work, we hypothesize that, in order to ‘fairly’ evalu- ate MT systems based on different paradigms, simi- larities at more abstract linguistic levels must be an- alyzed. For that purpose, we have compiled a rich set of metrics operating at the lexical, syntactic and shallow-semantic levels (see Section 2). We present a comparative study on the behavior of several met- ric representatives from each linguistic level in the context of some of the cases reported by Koehn and Monz (2006) and Callison-Burch et al. (2006) (see Section 3). We show that metrics based on deeper linguistic information (syntactic/shallow-semantic) are able to produce more reliable system rankings than those …

Content truncated to 500 words