BLANC: Learning Evaluation Metrics for MT

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 740–747, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics BLANC1: Learning Evaluation Metrics for MT Lucian Vlad Lita and Monica Rogati and Alon Lavie Carnegie Mellon University {llita,mrogati,alavie}@cs.cmu.edu Abstract We introduce BLANC, a family of dy- namic, trainable evaluation metrics for ma- chine translation. Flexible, parametrized models can be learned from past data and automatically optimized to correlate well with human judgments for different cri- teria (e.g. adequacy, fluency) using dif- ferent correlation measures. Towards this end, we discuss ACS (all common skip- ngrams), a practical algorithm with train- able parameters that estimates reference- candidate translation overlap by comput- ing a weighted sum of all common skip- ngrams in polynomial time. We show that the BLEU and ROUGE metric families are special cases of BLANC, and we compare correlations with human judgments across these three metric families. We analyze the algorithmic complexity of ACS and argue that it is more powerful in modeling both local meaning and sentence-level structure, while offering the same practicality as the established algorithms it generalizes. 1 Introduction Although recent MT evaluation methods show promising correlations to human judgments in terms of adequacy and fluency, there is still considerable room for improvement (Culy and Riehemann, 2003). Most of these studies have been performed at a sys- tem level and have not investigated metric robust- ness at a lower granularity. Moreover, even though the emphasis on adequacy vs. fluency is application- dependent, automatic evaluation metrics do not dis- tinguish between the need to optimize correlation with regard to one or the other. Machine translation automatic evaluation metrics face two important challenges: the lack of powerful features to capture both sentence level structure and local meaning, and the difficulty of designing good functions for combining these features into meaning- ful quality estimation algorithms. In this paper, we introduce BLANC1, an automatic MT evaluation metric family that is a generaliza- tion of popular and successful metric families cur- rently used in the MT community (BLEU, ROUGE, F- measure etc.). We describe an efficient, polynomial- time algorithm for BLANC, and show how it can be optimized to target adequacy, fluency or any other criterion. We compare our metric’s performance with traditional and recent automatic evaluation met- rics. We also describe the parameter conditions under which BLANC can emulate them. Throughout the remainder of this paper, we dis- tinguish between two components of automatic MT evaluation: the statistics computed on candidate and reference translations and the function used in defining evaluation metrics and generating transla- tion scores. Commonly used statistics include bag- of-words overlap, edit distance, longest common sub- sequence, ngram overlap, and skip-bigram overlap. Preferred functions are various combinations of pre- cision and recall (Soricut and Brill, 2004), including 1Since existing evaluation metrics (e.g. BLEU, ROUGE) are special cases of our metric family, it is only natural to name it Broad Learning and Adaptation for Numeric Criteria (BLANC) – white light contains light …

Content truncated to 500 words