The Next Step for Multi-Document Summarization: A Heterogeneous Multi-Genre Corpus Built with a Novel Construction Approach

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1535–1545, Osaka, Japan, December 11-17 2016. The Next Step for Multi-Document Summarization: A Heterogeneous Multi-Genre Corpus Built with a Novel Construction Approach Markus Zopf, Maxime Peyrard and Judith Eckle-Kohler Research Training Group AIPHES / Knowledge Engineering Group / UKP Lab Department of Computer Science, Technische Universit¨at Darmstadt Hochschulstraße 10, 64289 Darmstadt, Germany www.aiphes.tu-darmstadt.de, www.ukp.tu-darmstadt.de Abstract Research in multi-document summarization has focused on newswire corpora since the early beginnings. However, the newswire genre provides genre-specific features such as sentence po- sition which are easy to exploit in summarization systems. Such easy to exploit genre-specific features are available in other genres as well. We therefore present the new hMDS corpus for multi-document summarization, which contains heterogeneous source documents from multiple text genres, as well as summaries with different lengths. For the construction of the corpus, we developed a novel construction approach which is suited to build large and heterogeneous summarization corpora with little effort. The method reverses the usual process of writing sum- maries for given source documents: it combines already available summaries with appropriate source documents. In a detailed analysis, we show that our new corpus is significantly differ- ent from the homogeneous corpora commonly used, and that it is heterogeneous along several dimensions. Our experimental evaluation using well-known state-of-the-art summarization sys- tems shows that our corpus poses new challenges in the field of multi-document summarization. Last but not least, we make our corpus publicly available to the research community at the corpus web page https://github.com/AIPHES/hMDS. 1 Introduction Multi-document summarization (MDS) is the task of creating a summary from a topically related docu- ment collection. Existing corpora for the evaluation of MDS systems, most notably from the Document Understanding Conference (DUC) (Over et al., 2007) and from the Text Analysis Conference (TAC),1 cover mostly MDS of news documents (Nenkova et al., 2011). While research on MDS has also con- sidered genres other than newswire (e.g., opinionated blog posts in TAC 2008 or biomedical research papers in TAC 2014), MDS has almost exclusively focused on homogeneous document collections that belong to the same genre. However, this homogeneous nature of the existing MDS benchmark corpora does not reflect appli- cation scenarios where topically related documents from different genres need to be summarized. An exemplary scenario of increasing importance is MDS on the web, where a user retrieves multiple online documents for a particular topic (cf. Rosner and Camilleri (2008), Nenkova and McKeown (2011)). These online documents may comprise news articles, blog posts, encyclopedic texts, or even scientific articles. A related issue is the very high effort needed to create a new MDS corpus. Summarizing a set of documents is a demanding task for humans and requires expert abstractors, e.g., information analysts as in the DUC competitions. Attempts to obtain human-written summaries via crowdsourcing have failed (Lloret et al., 2013). When the set of documents to be summarized grows, the summarization task even becomes infeasible for humans. We address these …

Content truncated to 500 words