Tolerant BLEU: a Submission to the WMT14 Metrics Task

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 409–413, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Tolerant BLEU: a Submission to the WMT14 Metrics Task Jindˇrich Libovick´y and Pavel Pecina Charles University in Prague, Faculty of Mathematics and Physics Institute of Formal ad Applied Linguistics {libovicky, pecina}@ufal.mff.cuni.cz Abstract This paper describes a machine translation metric submitted to the WMT14 Metrics Task. It is a simple modification of the standard BLEU metric using a monolin- gual alignment of reference and test sen- tences. The alignment is computed as a minimum weighted maximum bipartite matching of the translated and the refer- ence sentence words with respect to the relative edit distance of the word prefixes and suffixes. The aligned words are in- cluded in the n-gram precision compu- tation with a penalty proportional to the matching distance. The proposed tBLEU metric is designed to be more tolerant to errors in inflection, which usually does not effect the understandability of a sentence, and therefore be more suitable for measur- ing quality of translation into morphologi- cally richer languages. 1 Introduction Automatic evaluation of machine translation (MT) quality is an important part of the machine trans- lation pipeline. The possibility to run an evalua- tion algorithm many times while training a system enables the system to be optimized with respect to such a metric (e.g., by Minimum Error Rate Train- ing (Och, 2003)). By achieving a high correlation of the metric with human judgment, we expect the system performance to be optimized also with re- spect to the human perception of translation qual- ity. In this paper, we propose an MT metric called tBLEU (tolerant BLEU) that is based on the stan- dard BLEU (Papineni et al., 2002) and designed to suit better when translation into morphologically richer languages. We aim to have a simple lan- guage independent metric that correlates with hu- man judgment better than the standard BLEU. Several metrics try to address this problem as well and usually succeed to gain a higher correlation with human judgment (e.g. ME- TEOR (Denkowski and Lavie, 2011), TerrorCat (Fishel et al., 2012)). However, they usually use some language-dependent tools and resources (METEOR uses stemmer and parahprasing tables, TerrorCat uses lemmatization and needs training data for each language pair) which prevent them from being widely adopted. In the next section, the previous work is briefly summarized. Section 3 describes the metric in de- tail. The experiments with the metric are described in Section 4 and their results are summarized in Section 5. 2 Previous Work BLEU (Papineni et al., 2002) is an established and the most widely used automatic metric for evalua- tion of MT quality. It is computed as a harmonic mean of the n-gram precisions multiplied by the brevity penalty coefficient which ensures also high recall. Formally: BLEU = BP · exp � 4 � n=1 1 4 log pn � , where BP is the brevity penaly defined as follows: BP = � 1 if c …

Content truncated to 500 words