Evaluation without references: IBM1 scores as evaluation metrics

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 99–103, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Evaluation without references: IBM1 scores as evaluation metrics Maja Popovi´c, David Vilar, Eleftherios Avramidis, Aljoscha Burchardt German Research Center for Artificial Intelligence (DFKI) Language Technology (LT), Berlin, Germany name.surname@dfki.de Abstract Current metrics for evaluating machine trans- lation quality have the huge drawback that they require human-quality reference transla- tions. We propose a truly automatic evalua- tion metric based on IBM1 lexicon probabili- ties which does not need any reference transla- tions. Several variants of IBM1 scores are sys- tematically explored in order to find the most promising directions. Correlations between the new metrics and human judgments are cal- culated on the data of the third, fourth and fifth shared tasks of the Statistical Machine Trans- lation Workshop. Five different European lan- guages are taken into account: English, Span- ish, French, German and Czech. The results show that the IBM1 scores are competitive with the classic evaluation metrics, the most promising being IBM1 scores calculated on morphemes and POS-4grams. 1 Introduction Currently used evaluation metrics such as BLEU (Pa- pineni et al., 2002), METEOR (Banerjee and Lavie, 2005), etc. are based on the comparison between human reference translations and the automatically generated hypotheses in the target language to be evaluated. While this scenario helps in the design of machine translation systems, it has two major drawbacks. The first one is the practical criticism that using reference translations is inefficient and ex- pensive: in real-life situations, the quality of ma- chine translation must be evaluated without having to pay humans for producing reference translations first. The second criticism is methodological: in using reference translation, the problem of evalu- ating translation quality (e.g., completeness, order- ing, domain fit, etc.) is transformed into a kind of paraphrase evaluation in the target language, which is a very difficult problem itself. In addition, the set of selected references always represents only a small subset of all good translations. To remedy these drawbacks, we propose a truly automatic eval- uation metric which is based on the IBM1 lexicon scores (Brown et al., 1993). The inclusion of IBM1 scores in translation sys- tems has shown experimentally to improve transla- tion quality (Och et al., 2003). They also have been used for confidence estimation for machine transla- tion (Blatz et al., 2003). To the best of our knowl- edge, these scores have not yet been used as an eval- uation metric. We carry out a systematic comparison between several variants of IBM1 scores. The Spearman’s rank correlation coefficients on the document (sys- tem) level between the IBM1 metrics and the hu- man ranking are computed on the English, French, Spanish, German and Czech texts generated by var- ious translation systems in the framework of the third (Callison-Burch et al., 2008), fourth (Callison- Burch et al., 2009) and fifth (Callison-Burch et al., 2010) shared translation tasks. 2 IBM1 scores The IBM1 model is a bag-of-word translation model which …

Content truncated to 500 words