Morphemes and POS tags for n-gram based evaluation metrics

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 104–107, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Morphemes and POS tags for n-gram based evaluation metrics Maja Popovi´c German Research Center for Artificial Intelligence (DFKI) Language Technology (LT), Berlin, Germany maja.popovic@dfki.de Abstract We propose the use of morphemes for auto- matic evaluation of machine translation out- put, and systematically investigate a set of F score and BLEU score based metrics calculated on words, morphemes and POS tags along with all corresponding combinations. Correlations between the new metrics and human judg- ments are calculated on the data of the third, fourth and fifth shared tasks of the Statisti- cal Machine Translation Workshop. Machine translation outputs in five different European languages are used: English, Spanish, French, German and Czech. The results show that the F scores which take into account morphemes and POS tags are the most promising metrics. 1 Introduction Recent investigations have shown that the n-gram based evaluation metrics calculated on Part-of- Speech (POS) sequences correlate very well with human judgments (Callison-Burch et al., 2008; Callison-Burch et al., 2009; Popovi´c and Ney, 2009) clearly outperforming the widely used metrics BLEU and TER. The BLEU score measured on morphemes is shown to be useful for evaluation of morpholog- ically rich languages (Luong et al., 2010). We pro- pose the use of morphemes for a set of n-gram based automatic evaluation metrics and investigate the cor- relation of the novel metrics with human judgments. We carry out a systematic comparison between the F and BLEU based metrics calculated on various combinations of words, morphemes and POS tags. The focus of this work is not a comparison of the morpheme and POS based metrics with the standard evaluation metrics1 as in (Popovi´c and Ney, 2009), but rather a comparison within the proposed set of metrics in order to decide which score(s) should be submitted to the WMT 2011 evaluation task. There are fifteen evaluation metrics in total, which can be divided in three groups: the metrics calculated on single units, i.e. words, morphemes or POS tags alone, the metrics calculated on pairs, i.e. words and POS tags, words and morphemes as well as mor- phemes and POS tags, and the metrics which take ev- erything into account – lexical, morphological and syntactic information, i.e. words, morphemes and POS tags. Spearman’s rank correlation coefficients on the document (system) level between all the metrics and the human ranking are computed on the En- glish, French, Spanish, German and Czech texts generated by various translation systems in the framework of the third (Callison-Burch et al., 2008), fourth (Callison-Burch et al., 2009) and fifth (Callison-Burch et al., 2010) shared translation tasks. 2 Evaluation metrics We carried out a systematic comparison between the following metrics: • single unit (word/morpheme/POS) metrics: – WORDF Standard F score: takes into account all word n-grams which have a counterpart 1Apart from the standard BLEU score which is tightly re- lated. 104 both in the corresponding reference and …

Content truncated to 500 words