Corroborating Text Evaluation Results with Heterogeneous Measures

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 455–466, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Corroborating Text Evaluation Results with Heterogeneous Measures Enrique Amig´o † Julio Gonzalo † Jes´us Gim´enez ‡ Felisa Verdejo† † UNED, Madrid {enrique,julio,felisa}@lsi.uned.es ‡ UPC, Barcelona {jgimenez}@lsi.upc.edu Abstract Automatically produced texts (e.g. transla- tions or summaries) are usually evaluated with n-gram based measures such as BLEU or ROUGE, while the wide set of more sophisti- cated measures that have been proposed in the last years remains largely ignored for practical purposes. In this paper we first present an in- depth analysis of the state of the art in order to clarify this issue. After this, we formalize and verify empirically a set of properties that every text evaluation measure based on simi- larity to human-produced references satisfies. These properties imply that corroborating sys- tem improvements with additional measures always increases the overall reliability of the evaluation process. In addition, the greater the heterogeneity of the measures (which is mea- surable) the higher their combined reliability. These results support the use of heterogeneous measures in order to consolidate text evalua- tion results. 1 Introduction The automatic evaluation of textual outputs is a core issue in many Natural Language Processing (NLP) tasks such as Natural Language Generation, Machine Translation (MT) and Automatic Sum- marization (AS). State-of-the-art automatic evalu- ation methods all operate by rewarding similari- ties between automatically-produced candidate out- puts and manually-produced reference solutions, so- called human references or models. Over the last decade, a wide variety of measures, based on different quality assumptions, have been proposed. Recent work suggests exploiting exter- nal knowledge sources and/or deep linguistic an- notation, and measure combination (see Section 2). However, original measures based on lexical match- ing, such as BLEU (Papineni et al., 2001a) and ROUGE (Lin, 2004) are still preferred as de facto standards in MT and AS, respectively. There are, in our opinion, two main reasons behind this fact. First, the use of a common measure certainly allows researchers to carry out objective comparisons be- tween their work and other published results. Sec- ond, the advantages of novel measures are not easy to demonstrate in terms of correlation with human judgements. Our goal is not to answer which is the most re- liable metric or to propose yet another novel mea- sure. Rather than this, we first analyze in depth the state of the art, concluding that it is not easy to de- termine the reliability of a measure. In absence of a clear proof of the advantages of novel measures, sys- tem developers naturally tend to prefer well-known standard measures. Second, we formalize and check empirically two intrinsic properties that any evalua- tion measure based on similarity to human-produced references satisfies. Assuming that a measure satis- fies a set of basic formal constraints, these properties imply that corroborating a system comparison with additional measures always increases the overall re- liability of the evaluation process, even when the …

Content truncated to 500 words