Syntax-Oriented Evaluation Measures for Machine Translation Output

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 29–32, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics Syntax-oriented evaluation measures for machine translation output Maja Popovi´c and Hermann Ney RWTH Aachen University Aachen, Germany popovic,ney@informatik.rwth-aachen.de Abstract We explored novel automatic evaluation measures for machine translation output oriented to the syntactic structure of the sentence: the BLEU score on the detailed Part-of-Speech (POS) tags as well as the precision, recall and F-measure obtained on POS n-grams. We also introduced F- measure based on both word and POS n- grams. Correlations between the new met- rics and human judgments were calcu- lated on the data of the first, second and third shared task of the Statistical Machine Translation Workshop. Machine transla- tion outputs in four different European languages were taken into account: En- glish, Spanish, French and German. The results show that the new measures cor- relate very well with the human judge- ments and that they are competitive with the widely used BLEU, METEOR and TER metrics. 1 Introduction We proposed several syntax-oriented automatic evaluation measures based on sequences of POS tags and investigated how they correlate with hu- man judgments. The new measures are the POS- BLEU score, i.e. the BLEU score calculated on POS tags instead of words, as well as the POSP, the POSR and the POSF score: precision, recall and F- measure calculated on POS n-grams. In addition to the metrics based only on POS tags, we investi- gated a WPF score, i.e. an F-measure which takes into account both word and POS n-grams. The correlations on the document level were computed on the English, French, Spanish and German texts generated by various translation sys- tems in the framework of the first (Koehn and Monz, 2006), second (Callison-Burch et al., 2007) and third shared translation task (Callison-Burch et al., 2008). Preliminary experiments were car- ried out on the data from the first (2006) and the second task (2007) – Spearman’s rank corre- lation coefficients between the adequacy and flu- ency scores and the POSBLEU, POSP, POSR and POSF scores were calculated. The POSBLEU and the POSF score were shown to be the most promis- ing, so that these metrics were submitted to the official shared evaluation task 2008. The results of this evaluation showed that these metrics also correlate well on the document level with another human score, i.e. the sentence ranking. However, on the sentence level the results were less promis- ing. The possible reason for this is the main draw- back of the metrics based on pure POS tags, i.e. neglecting the lexical aspect. Therefore we also introduced a WPF score which takes into account both word n-grams and POS n-grams. 2 Syntactic-oriented evaluation metrics We investigated the following metrics oriented on the syntactic structure of a translation output: • POSBLEU The standard BLEU score (Papineni et al., 2002) calculated on POS tags instead of words; • POSP POS n-gram precision: percentage of POS n- grams in …

Content truncated to 500 words