Evaluation Challenges in Large-Scale Document Summarization

Evaluation challenges in large-scale document summarization Dragomir R. Radev U. of Michigan radev@umich.edu Wai Lam Chinese U. of Hong Kong wlam@se.cuhk.edu.hk Arda C¸ elebi USC/ISI ardax@isi.edu Simone Teufel U. of Cambridge simone.teufel@cl.cam.ac.uk John Blitzer U. of Pennsylvania blitzer@seas.upenn.edu Danyu Liu U. of Alabama liudy@cis.uab.edu Horacio Saggion U. of Sheffield h.saggion@dcs.shef.ac.uk Hong Qi U. of Michigan hqi@umich.edu Elliott Drabek Johns Hopkins U. edrabek@cs.jhu.edu Abstract We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of (a) 100 Million auto- matic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese, (b) more than 10,000 manual abstracts and extracts, and (c) 200 Million automatic document and summary retrievals using 20 queries. We present both qualitative and quantitative results showing the strengths and draw- backs of all evaluation methods and how they rank the different summarizers. 1 Introduction Automatic document summarization is a field that has seen increasing attention from the NLP commu- nity in recent years. In part, this is because sum- marization incorporates many important aspects of both natural language understanding and natural lan- guage generation. In part it is because effective auto- matic summarization would be useful in a variety of areas. Unfortunately, evaluating automatic summa- rization in a standard and inexpensive way is a diffi- cult task (Mani et al., 2001). Traditional large-scale evaluations are either too simplistic (using measures like precision, recall, and percent agreement which (1) don’t take chance agreement into account and (2) don’t account for the fact that human judges don’t agree which sentences should be in a summary) or too expensive (an approach using manual judge- ments can scale up to a few hundred summaries but not to tens or hundreds of thousands). In this paper, we present a comparison of six summarizers as well as a meta-evaluation including eight measures: Precision/Recall, Percent Agree- ment, Kappa, Relative Utility, Relevance Correla- tion, and three types of Content-Based measures (cosine, longest common subsequence, and word overlap). We found that while all measures tend to rank summarizers in different orders, measures like Kappa, Relative Utility, Relevance Correlation and Content-Based each offer significant advantages over the more simplistic methods. 2 Data, Annotation, and Experimental Design We performed our experiments on the Hong Kong News corpus provided by the Hong Kong SAR of the People’s Republic of China (LDC catalog num- ber LDC2000T46). It contains 18,146 pairs of par- allel documents in English and Chinese. The texts are not typical news articles. The Hong Kong News- paper mainly publishes announcements of the local administration and descriptions of municipal events, such as an anniversary of the fire department, or sea- sonal festivals. We tokenized the corpus to iden- tify headlines and sentence boundaries. For the En- glish text, we used a lemmatizer for nouns and verbs. We also segmented the Chinese documents using the tool provided at http://www.mandarintools.com. Several steps of the meta evaluation that we per- formed involved human annotator …

Content truncated to 500 words