Beyond Generic Summarization: A Multi-faceted Hierarchical Summarization Corpus of Large Heterogeneous Data

Beyond Generic Summarization: A Multi-faceted Hierarchical Summarization Corpus of Large Heterogeneous Data Christopher Tauchmann⋆, Thomas Arnold⋆, Andreas Hanselowski⋆, Christian M. Meyer⋆ and Margot Mieskes‡ Research Training Group AIPHES ⋆Technische Universit¨at Darmstadt; ‡Hochschule Darmstadt https://www.aiphes.tu-darmstadt.de Abstract Automatic summarization has so far focused on datasets of ten to twenty rather short documents, typically news articles. But automatic systems could in theory analyze hundreds of documents from a wide range of sources and provide an overview to the interested reader. Such a summary would ideally present the most general issues of a given topic and allow for more in-depth information on specific aspects within said topic. In this paper, we present a new approach for creating hierarchical summarization corpora from large, heterogeneous document collections. We first extract relevant content using crowdsourcing and then ask trained annotators to order the relevant information hierarchically. This yields tree structures covering the specific facets discussed in a document collection. Our resulting corpus is freely available and can be used to develop and evaluate hierarchical summarization systems. Keywords: hierarchical summarization, large corpora, heterogeneous sources, crowdsourcing, aspect-oriented summarization 1. Introduction Automatically created summaries are most useful if they allow readers to save time when reading long and/or many documents from a large number of sources. However, many state-of-the-art approaches in automatic multi-document summarization (MDS) are still evaluated on small clus- ters of ten to twenty short articles. The most prominent document collections from the DUC and TAC conferences have, for example, only about 6,700 (DUC ’04) and 17,400 (DUC ’06) tokens per topic cluster.1 This evaluation setup does not cover the full potential of automatic summariza- tion, which could easily aggregate collections of over hun- dred documents with more than 100,000 tokens. In some respects, the current setup is not even very real- istic, as the vast majority of the available datasets cover only newswire text about a single event or entity (Nenkova, 2005). Given the large amount of redundancy in this text type, a human reader could read only one or two of the source documents and quickly skim over the remaining ones to get a good overview of the article’s main event or entity – albeit update summaries would be helpful in this situation. Even more recent work in social media and real-time summarization is based on high-redundancy text (Chua and Asur, 2013; Lin et al., 2016). In large heteroge- neous document collections, there are important facts and arguments that appear only in few of the available docu- ments and are therefore missed by generic summary strate- gies and absent from both automatic and reference sum- maries. With increasing volume, velocity, and variety of the source documents, it gets, however, extremely difficult to construct suitable evaluation corpora. Assuming a reading speed of 228 ± 30 words per minute for English (Trauzettel- Klosinski and Dietz, 2012), it already takes more than seven hours (excluding breaks) to read a document collection with 1http://duc.nist.gov, http://tac.nist.gov 100,000 words. It is hardly possible for an individual an- notator to stay equally concentrated …

Content truncated to 500 words