LeBLEU: N-gram-based Translation Evaluation Score for Morphologically Complex Languages

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 411–416, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. LeBLEU: N-gram-based Translation Evaluation Score for Morphologically Complex Languages Sami Virpioja Lingsoft, Inc. Helsinki, Finland sami.virpioja@lingsoft.fi Stig-Arne Gr¨onroos Aalto University Dept. of Signal Processing and Acoustics stig-arne.gronroos@aalto.fi Abstract This paper describes the LeBLEU evalu- ation score for machine translation, sub- mitted to WMT15 Metrics Shared Task. LeBLEU extends the popular BLEU score to consider fuzzy matches between word n-grams. While there are several variants of BLEU that allow to non-exact matches between words either by character-based distance measures or morphological pre- processing, none of them use fuzzy com- parison between longer chunks of text. The results on WMT data sets show that fuzzy n-gram matching improves correla- tions to human evaluation especially for highly compounding languages. 1 Introduction The quality of machine translation has improved to the level that the translation hypotheses are useful starting points for human translators for almost any language pair. In the post-editing task, the ultimate way to evaluate the machine translation quality is to measure the editing time. Editing times are naturally related to the num- ber and types of the edits—and thus the number of keystrokes (Frederking and Nirenburg, 1994)— the post-editor needs to get the final translation from the hypothesis. If we compare the raw trans- lation hypothesis and its post-edited version, an appropriate edit distance measure should correlate to the edit time. However, implementing such a measure is far from trivial. In automatic speech recognition, common eval- uation measures are Word Error Rate (WER) and Letter Error Rate (LER) that are based on the Lev- enshtein edit distance (Levenshtein, 1966). LER is more reasonable measure than WER for morpho- logically complex languages, in which the same word can occur in many inflected and derived forms (Creutz et al., 2007). However, both give too high penalty for the variations in word order- ing, which are frequent in translations. Even in English, there are often at least two grammatically correct orders for a complex sentence. For lan- guages in which the grammatical roles are marked by morphology and not the word order, there may be many more options. An edit distance measure suitable for machine translation would require move operations. How- ever, such measures are computationally very ex- pensive: finding the minimum edit distance with moves is NP-hard (Shapira and Storer, 2002), making it cumbersome for evaluation and unsuit- able for automatic tuning of the translation mod- els. Possible solutions include limiting the move operations or searching only for an approximate solution. For example, Translation Edit Rate (TER) by Snover et al. (2006) uses a shift oper- ation that moves a contiguous sequence of words to another location, as well as a greedy search al- gorithm to find the minimum distance. Stanford Probabilistic Edit Distance Evaluation (SPEDE) by Wang and Manning (2012) applies a proba- bilistic push-down automaton that captures non- nested, limited distance word swapping. A different approach to avoid the requirement …

Content truncated to 500 words