Comparing Automatic and Human Evaluation of NLG Systems

Comparing Automatic and Human Evaluation of NLG Systems Anja Belz Natural Language Technology Group CMIS, University of Brighton UK A.S.Belz@brighton.ac.uk Ehud Reiter Dept of Computing Science University of Aberdeen UK ereiter@csd.abdn.ac.uk Abstract We consider the evaluation problem in Natural Language Generation (NLG) and present results for evaluating several NLG systems with similar functionality, includ- ing a knowledge-based generator and sev- eral statistical systems. We compare eval- uation results for these systems by human domain experts, human non-experts, and several automatic evaluation metrics, in- cluding NIST, BLEU, and ROUGE. We find that NIST scores correlate best (> 0.8) with human judgments, but that all automatic metrics we examined are bi- ased in favour of generators that select on the basis of frequency alone. We con- clude that automatic evaluation of NLG systems has considerable potential, in par- ticular where high-quality reference texts and only a small number of human evalua- tors are available. However, in general it is probably best for automatic evaluations to be supported by human-based evaluations, or at least by studies that demonstrate that a particular metric correlates well with hu- man judgments in a given domain. 1 Introduction Evaluation is becoming an increasingly important topic in Natural Language Generation (NLG), as in other fields of computational linguistics. Some NLG researchers are impressed by the success of the BLEU evaluation metric (Papineni et al., 2002) in Machine Translation (MT), which has trans- formed the MT field by allowing researchers to quickly and cheaply evaluate the impact of new ideas, algorithms, and data sets. BLEU and re- lated metrics work by comparing the output of an MT system to a set of reference (‘gold standard’) translations, and in principle this kind of evalua- tion could be done with NLG systems as well. In- deed NLG researchers are already starting to use BLEU (Habash, 2004; Belz, 2005) in their evalua- tions, as this is much cheaper and easier to organ- ise than the human evaluations that have tradition- ally been used to evaluate NLG systems. However, the use of such corpus-based evalua- tion metrics is only sensible if they are known to be correlated with the results of human-based eval- uations. While studies have shown that ratings of MT systems by BLEU and similar metrics corre- late well with human judgments (Papineni et al., 2002; Doddington, 2002), we are not aware of any studies that have shown that corpus-based evalu- ation metrics of NLG systems are correlated with human judgments; correlation studies have been made of individual components (Bangalore et al., 2000), but not of systems. In this paper we present an empirical study of how well various corpus-based metrics agree with human judgments, when evaluating several NLG systems that generate sentences which de- scribe changes in the wind (for weather forecasts). These systems do not perform content determina- tion (they are limited to microplanning and realisa- tion), so our study does not address corpus-based evaluation of content determination. 2 Background 2.1 Evaluation of NLG systems NLG systems have traditionally …

Content truncated to 500 words