The TIPSTER SUMMAC Text Summarization Evaluation

Proceedings of EACL '99 The TIPSTER SUMMAC Text Summarization Evaluation Inderjeet Mani David House Gary Klein Lynette Hirschman* The MITRE Corporation 11493 Sunset Hills Rd. Reston, VA 22090 USA Therese Firmin Department of Defense 9800 Savage Rd. Ft. Meade, MD 20755 USA Beth Sundheim SPAWAR Systems Center Code D44208 53140 Gatchell Rd. San Diego, CA 92152 USA Abstract The TIPSTER Text Summarization Evaluation (SUMMAC) has established definitively that automatic text summa- rization is very effective in relevance as- sessment tasks. Summaries as short as 17% of full text length sped up decision- making by almost a factor of 2 with no statistically significant degradation in F- score accuracy. SUMMAC has also in- troduced a new intrinsic method for au- tomated evaluation of informative sum- maries. 1 Introduction In May 1998, the U.S. government completed the TIPSTER Text Summarization Evaluation (SUMMAC), which was the first large-scale, developer-independent evaluation of automatic text summarization systems. The goals of the SUMMAC evaluation were to judge individual summarization systems in terms of their useful- ness in specific summarization tasks and to gain a better understanding of the issues involved in building and evaluating such systems. 1.1 Text Summarization Text summarization is the process of distilling the most important information from a set of sources to produce an abridged version for particular users and tasks (Maybury 1995). Since abridgment is crucial, an important parameter to summariza- tion is the level of compression (ratio of summary length to source length) desired. Summaries can be used to indicate what topics are addressed in the source text, and thus can be used to alert the user as to source content (the indicative function). In addition, summaries can also be used to stand in place of the source (the informative function). 202 Burlington Rd.,' Bedford, MA 01730 They can even offer a critique of the source (the evaluative function) (Sparck-Jones 1998). Often, summaries are tailored to a reader's interests and expertise, yielding topic-relatedsummaries, or else they can be aimed at a broad readership com- munity, as in the case of generic summaries. It is also useful to distinguish between summaries which are extracts of source material, and those which are abstracts containing new text generated by the summarizer. 1.2 Summarization Evaluation Methods Methods for evaluating text summarization can be broadly classified into two categories. The first, an intrinsic (or normative) evalua- tion, judges the quality of the summary directly based on analysis in terms of some set of norms. This can involve user judgments of fluency of the summary (Minel et al. 1997), (Brandow et al. 1994), coverage of stipulated "key/essential ideas" in the source (Paice 1990), (Brandow et al. 1994), or similarity to an "ideal" summary, e.g., (Ed- mundson 1969), (Kupiec et al. 1995). The problem with matching a system summary against an ideal summary is that the ideal sum- mary is hard to establish. There can be a large number of generic and topic-related abstracts that could summarize a given document. Also, there have been several reports of …

Content truncated to 500 words