Evaluating Content Selection in Summarization: The Pyramid Method

Evaluating Content Selection in Summarization: The Pyramid Method Ani Nenkova and Rebecca Passonneau Columbia University Computer Science Department New York, NY 10027 {ani,becky}@cs.columbia.edu Abstract We present an empirically grounded method for evaluating content selection in summariza- tion. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative im- portance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus im- proves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference. 1 Introduction Evaluating content selection in summarization has proven to be a difficult problem. Our approach acknowledges the fact that no single best model summary exists, and takes this as a foundation rather than an obstacle. In ma- chine translation, the rankings from the automatic BLEU method (Papineni et al., 2002) have been shown to corre- late well with human evaluation, and it has been widely used since and has even been adapted for summarization (Lin and Hovy, 2003). To show that an automatic method is a reasonable approximation of human judgments, one needs to demonstrate that these can be reliably elicited. However, in contrast to translation, where the evaluation criterion can be defined fairly precisely it is difficult to elicit stable human judgments for summarization (Rath et al., 1961) (Lin and Hovy, 2002). Our approach tailors the evaluation to observed dis- tributions of content over a pool of human summaries, rather than to human judgments of summaries. Our method involves semantic matching of content units to which differential weights are assigned based on their fre- quency in a corpus of summaries. This can lead to more stable, more informative scores, and hence to a meaning- ful content evaluation. We create a weighted inventory of Summary Content Units–a pyramid–that is reliable, pre- dictive and diagnostic, and which constitutes a resource for investigating alternate realizations of the same mean- ing. No other evaluation method predicts sets of equally informative summaries, identifies semantic differences between more and less highly ranked summaries, or con- stitutes a tool that can be applied directly to further anal- ysis of content selection. In Section 2, we describe the DUC method. In Sec- tion 3 we present an overview of our method, contrast our scores with other methods, and describe the distribu- tion of scores as pyramids grow in size. We compare our approach with previous work in Section 4. In Section 5, we present our conclusions and point to our next step, the feasibility of automating our method. A more detailed account of the work described here, but not including the study of distributional properties of pyramid scores, can be found in (Passonneau and Nenkova, 2003). 2 Current Approach: the Document Understanding Conference 2.1 DUC Within DUC, different types of summarization have been studied: the generation of abstracts and extracts of differ- ent lengths, single- and multi-document summaries, and summaries focused by topic or opinion. Evaluation in- volves comparison of a peer summary …

Content truncated to 500 words