Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics

Automatic Evaluation of Summaries Using N-gram Co-Occurrence Statistics Chin-Yew Lin and Eduard Hovy Information Sciences Institute University of Southern California 4676 Admiralty Way Marina del Rey, CA 90292 {cyl,hovy}@isi.edu Abstract Following the recent adoption by the machine translation community of automatic evalua- tion using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co- occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct ap- plication of the BLEU evaluation procedure does not always give good results. 1 Introduction Automated text summarization has drawn a lot of inter- est in the natural language processing and information retrieval communities in the recent years. A series of workshops on automatic text summarization (WAS 2000, 2001, 2002), special topic sessions in ACL, COLING, and SIGIR, and government sponsored evaluation efforts in the United States (DUC 2002) and Japan (Fukusima and Okumura 2001) have advanced the technology and produced a couple of experimental online systems (Radev et al. 2001, McKeown et al. 2002). Despite these efforts, however, there are no common, convenient, and repeatable evaluation meth- ods that can be easily applied to support system devel- opment and just-in-time comparison among different summarization methods. The Document Understanding Conference (DUC 2002) run by the National Institute of Standards and Technol- ogy (NIST) sets out to address this problem by provid- ing annual large scale common evaluations in text summarization. However, these evaluations involve human judges and hence are subject to variability (Rath et al. 1961). For example, Lin and Hovy (2002) pointed out that 18% of the data contained multiple judgments in the DUC 2001 single document evaluation1. To further progress in automatic summarization, in this paper we conduct an in-depth study of automatic evaluation methods based on n-gram co-occurrence in the context of DUC. Due to the setup in DUC, the evaluations we discussed here are intrinsic evaluations (Spärck Jones and Galliers 1996). Section 2 gives an overview of the evaluation procedure used in DUC. Section 3 discusses the IBM BLEU (Papineni et al. 2001) and NIST (2002) n-gram co-occurrence scoring procedures and the application of a similar idea in evaluating summaries. Section 4 compares n-gram co- occurrence scoring procedures in terms of their correla- tion to human results and on the recall and precision of statistical significance prediction. Section 5 concludes this paper and discusses future directions. 2 Document Understanding Conference The 2002 Document Understanding Conference2 in- cluded the follow two main tasks: • Fully automatic single-document summarization: given a document, participants were required to create a generic 100-word summary. The training set comprised 30 sets of approximately 10 docu- ments each, together with their 100-word human written summaries. The test set comprised 30 un- seen documents. • Fully automatic multi-document summarization: given a set of documents about a single subject, participants were required to create 4 generic sum- maries of the entire set, containing 50, 100, 200, and 400 words …

Content truncated to 500 words