Bleu: a Method for Automatic Evaluation of Machine Translation

BLEU: a Method for Automatic Evaluation of Machine Translation Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu IBM T. J. Watson Research Center Yorktown Heights, NY 10598, USA {papineni,roukos,toddward,weijing}@us.ibm.com Abstract Human evaluations of machine translation are extensive but expensive. Human eval- uations can take months to finish and in- volve human labor that can not be reused. We propose a method of automatic ma- chine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evalu- ation, and that has little marginal cost per run. We present this method as an auto- mated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations.1 1 Introduction 1.1 Rationale Human evaluations of machine translation (MT) weigh many aspects of translation, including ade- quacy, fidelity , and fluency of the translation (Hovy, 1999; White and O’Connell, 1994). A compre- hensive catalog of MT evaluation techniques and their rich literature is given by Reeder (2001). For the most part, these various human evaluation ap- proaches are quite expensive (Hovy, 1999). More- over, they can take weeks or months to finish. This is a big problem because developers of machine trans- lation systems need to monitor the effect of daily changes to their systems in order to weed out bad ideas from good ideas. We believe that MT progress stems from evaluation and that there is a logjam of fruitful research ideas waiting to be released from 1So we call our method the bilingual evaluation understudy, BLEU. the evaluation bottleneck. Developers would bene- fit from an inexpensive automatic evaluation that is quick, language-independent, and correlates highly with human evaluation. We propose such an evalua- tion method in this paper. 1.2 Viewpoint How does one measure translation performance? The closer a machine translation is to a professional human translation, the better it is. This is the cen- tral idea behind our proposal. To judge the quality of a machine translation, one measures its closeness to one or more reference human translations accord- ing to a numerical metric. Thus, our MT evaluation system requires two ingredients: 1. a numerical “translation closeness” metric 2. a corpus of good quality human reference trans- lations We fashion our closeness metric after the highly suc- cessful word error rate metric used by the speech recognition community, appropriately modified for multiple reference translations and allowing for le- gitimate differences in word choice and word or- der. The main idea is to use a weighted average of variable length phrase matches against the reference translations. This view gives rise to a family of met- rics using various weighting schemes. We have se- lected a promising baseline metric from this family. In Section 2, we describe the baseline metric in detail. In Section 3, we evaluate the performance of BLEU. In Section 4, we describe a human evaluation experiment. In Section 5, we compare our baseline metric performance with human evaluations. Computational Linguistics (ACL), Philadelphia, July 2002, pp. 311-318. Proceedings of the …

Content truncated to 500 words