Quantifying the Limits and Success of Extractive Summarization Systems Across Domains

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 903–911, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Quantifying the Limits and Success of Extractive Summarization Systems Across Domains Hakan Ceylan and Rada Mihalcea Department of Computer Science University of North Texas Denton, TX 76203 {hakan,rada}@unt.edu Umut ¨Ozertem Yahoo! Labs 701 First Avenue Sunnyvale, CA 94089 umut@yahoo-inc.com Elena Lloret and Manuel Palomar Department of Software and Computing Systems University of Alicante San Vicente del Raspeig Alicante 03690, Spain {elloret,mpalomar}@dlsi.ua.es Abstract This paper analyzes the topic identification stage of single-document automatic text sum- marization across four different domains, con- sisting of newswire, literary, scientific and le- gal documents. We present a study that ex- plores the summary space of each domain via an exhaustive search strategy, and finds the probability density function (pdf) of the ROUGE score distributions for each domain. We then use this pdf to calculate the per- centile rank of extractive summarization sys- tems. Our results introduce a new way to judge the success of automatic summarization systems and bring quantified explanations to questions such as why it was so hard for the systems to date to have a statistically signifi- cant improvement over the lead baseline in the news domain. 1 Introduction Topic identification is the first stage of the gener- ally accepted three-phase model in automatic text summarization, in which the goal is to identify the most important units in a document, i.e., phrases, sentences, or paragraphs (Hovy and Lin, 1999; Lin, 1999; Sparck-Jones, 1999). This stage is followed by the topic interpretation and summary generation steps where the identified units are further processed to bring the summary into a coherent, human read- able abstract form. The extractive summarization systems, however, only employ the topic identifi- cation stage, and simply output a ranked list of the units according to a compression ratio criterion. In general, for most systems sentences are the preferred units in this stage, as they are the smallest grammat- ical units that can express a statement. Since the sentences in a document are reproduced verbatim in extractive summaries, it is theoretically possible to explore the search space of this problem through an enumeration of all possible extracts for a document. Such an exploration would not only allow us to see how far we can go with extractive summarization, but we would also be able to judge the difficulty of the problem by looking at the dis- tribution of the evaluation scores for the generated extracts. Moreover, the high scoring extracts could also be used to train a machine learning algorithm. However, such an enumeration strategy has an exponential complexity as it requires all possible sentence combinations of a document to be gener- ated, constrained by a given word or sentence length. Thus the problem quickly becomes impractical as the number of sentences in a document increases and the compression ratio decreases. In this work, we try to overcome this bottleneck by using a large …

Content truncated to 500 words