Automatically Evaluating Content Selection in Summarization without Human Models

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 306–314, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Automatically Evaluating Content Selection in Summarization without Human Models Annie Louis University of Pennsylvania lannie@seas.upenn.edu Ani Nenkova University of Pennsylvania nenkova@seas.upenn.edu Abstract We present a fully automatic method for content selection evaluation in summariza- tion that does not require the creation of human model summaries. Our work capi- talizes on the assumption that the distribu- tion of words in the input and an informa- tive summary of that input should be sim- ilar to each other. Results on a large scale evaluation from the Text Analysis Con- ference show that input-summary compar- isons are very effective for the evaluation of content selection. Our automatic meth- ods rank participating systems similarly to manual model-based pyramid evalua- tion and to manual human judgments of responsiveness. The best feature, Jensen- Shannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations. 1 Introduction The most commonly used evaluation method for summarization during system development and for reporting results in publications is the auto- matic evaluation metric ROUGE (Lin, 2004; Lin and Hovy, 2003). ROUGE compares system sum- maries against one or more model summaries by computing n-gram word overlaps between the two. The wide adoption of such automatic mea- sures is understandable because they are conve- nient and greatly reduce the complexity of eval- uations. ROUGE scores also correlate well with manual evaluations of content based on compar- ison with a single model summary, as used in the early editions of the Document Understanding Conferences (Over et al., 2007). In our work, we take the idea of automatic evaluation to an extreme and explore the feasi- bility of developing a fully automatic evaluation method for content selection that does not make use of human model summaries at all. To this end, we show that evaluating summaries by comparing them with the input obtains good correlations with manual evaluations for both query focused and up- date summarization tasks. Our results have important implications for fu- ture development of summarization systems and their evaluation. High correlations between system ranking pro- duced with the fully automatic method and manual evaluations show that the new eval- uation measures can be used during system development when human model summaries are not available. Our results provide validation of several features that can be optimized in the development of new summarization systems when the objec- tive is to improve content selection on aver- age, over a collection of test inputs. However, none of the features is consistently predictive of good summary content for individual in- puts. We find that content selection performance on standard test collections can be approximated well by the proposed fully automatic method. This result greatly underlines the need to re- quire linguistic quality evaluations alongside content selection ones in future evaluations and research. 2 Model-free methods for evaluation Proposals for developing fully automatic methods for summary evaluation …

Content truncated to 500 words