A Repository of Corpora for Summarization

A Repository of Corpora for Summarization Franck Dernoncourt1, Mohammad Ghassemi2, Walter Chang1 1Adobe Research, 2MIT franck.dernoncourt@adobe.com, ghassemi@mit.edu, wachang@adobe.com Abstract Summarization corpora are numerous but fragmented, making it challenging for researchers to efficiently pinpoint corpora most suited to a given summarization task. In this paper, we introduce a repository containing corpora available to train and evaluate automatic summarization systems. We also present an overview of the main corpora with respect to the different summarization tasks, and identify various corpus parameters that researchers may want to consider when choosing a corpus. Lastly, as the recent successes of artificial neural networks for summarization have renewed the interest in creating large-scale corpora for summarization, we survey which corpora are used in neural network research studies. We come to the conclusion that more large-scale corpora for summarization are needed. Furthermore, each corpus is organized differently, which makes it time-consuming for researchers to experiment a new summarization algorithm on many corpora, and as a result studies typically use one or very few corpora. Agreeing on a data standard for summarization corpora would be beneficial to the field. Keywords: abstractive summarization, extractive summarization, artificial neural networks, corpora 1. Introduction Automatic summarization has been studied for over half a century (Luhn, 1958). Over the decades, many summa- rization tasks, systems, metrics, and corpora have been cre- ated. Summarization approaches may be categorized in ab- stractive, extractive, and compressive approaches. When the output of a summarization system is a newly generated text, distinct from the original document, it is referred to as abstractive. Systems that compose summaries by combin- ing and restructuring various segments of the original text, are referred to as extractive, or sentential extractive if sen- tences from the original text are selected to form the sum- mary. Lastly, systems that compose summaries by pruning tokens from the original text are referred to as compres- sive.1 Summarization systems may span single, or multiple- documents, and can produce outputs of varying lengths and structures. When the original text spans over multiple doc- uments, the task is called multi-document summarization. The length of the summaries differs across corpora: for ex- ample, sentence-level summarization aims at summarizing a text into a single sentence, typically abstractively, and headline generation aims at summarizing the text into a headline, which tends to be shorter than a sentence. Summarization is a subjective task (Rath et al., 1961; Lin and Hovy, 2002), requiring human input to assess perfor- mance. Generic summarization aims at creating a summary that is as reader-independent as possible, i.e. satisfying as many readers as possible. There has been some work on non-generic summarization, such as query-based and topic- based summarization, which bias the summary toward a query or a topic expressed by the intended reader (Hand, 1997). Automated evaluation methods have been developed. The most widely used automated evaluation metric for summa- rization is ROUGE and its variants (Lin and Hovy, 2003; 1Some researchers use the terms extractive and compressive interchangeably. Sentential extractive is unambiguous. Lin, 2004), followed by METEOR (Banerjee …

Content truncated to 500 words