Contextual Bitext-Derived Paraphrases in Automatic MT Evaluation

Proceedings of the Workshop on Statistical Machine Translation, pages 86–93, New York City, June 2006. c⃝2006 Association for Computational Linguistics Contextual Bitext-Derived Paraphrases in Automatic MT Evaluation Karolina Owczarzak Declan Groves Josef Van Genabith Andy Way National Centre for Language Technology School of Computing Dublin City University Dublin 9, Ireland {owczarzak,dgroves,josef,away}@computing.dcu.ie Abstract In this paper we present a novel method for deriving paraphrases during automatic MT evaluation using only the source and reference texts, which are necessary for the evaluation, and word and phrase alignment software. Using target language paraphrases produced through word and phrase alignment a number of alternative reference sentences are constructed auto- matically for each candidate translation. The method produces lexical and low- level syntactic paraphrases that are rele- vant to the domain in hand, does not use external knowledge resources, and can be combined with a variety of automatic MT evaluation system. 1 Introduction Since their appearance, BLEU (Papineni et al., 2002) and NIST (Doddington, 2002) have been the standard tools used for evaluating the quality of machine translation. They both score candidate translations on the basis of the number of n-grams it shares with one or more reference translations provided. Such automatic measures are indispen- sable in the development of machine translation systems, because they allow the developers to con- duct frequent, cost-effective, and fast evaluations of their evolving models. These advantages come at a price, though: an automatic comparison of n-grams measures only the string similarity of the candidate translation to one or more reference strings, and will penalize any divergence from them. In effect, a candidate translation expressing the source meaning accu- rately and fluently will be given a low score if the lexical choices and syntactic structure it contains, even though perfectly legitimate, are not present in at least one of the references. Necessarily, this score would not reflect a much more favourable human judgment that such a translation would re- ceive. The limitations of string comparison are the reason why it is advisable to provide multiple ref- erences for a candidate translation in the BLEU- or NIST-based evaluation in the first place. While (Zhang and Vogel, 2004) argue that increasing the size of the test set gives even more reliable system scores than multiple references, this still does not solve the inadequacy of BLEU and NIST for sen- tence-level or small set evaluation. On the other hand, in practice even a number of references do not capture the whole potential variability of the translation. Moreover, often it is the case that mul- tiple references are not available or are too difficult and expensive to produce: when designing a statis- tical machine translation system, the need for large amounts of training data limits the researcher to collections of parallel corpora like Europarl (Koehn, 2005), which provides only one reference, namely the target text; and the cost of creating ad- ditional reference translations of the test set, usu- ally a few thousand sentences long, often exceeds the resources available. Therefore, it would be …

Content truncated to 500 words