Bringing Structure into Summaries: Crowdsourcing a Benchmark Corpus of Concept Maps

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2951–2961 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Bringing Structure into Summaries: Crowdsourcing a Benchmark Corpus of Concept Maps Tobias Falke and Iryna Gurevych Research Training Group AIPHES and UKP Lab Department of Computer Science, Technische Universit¨at Darmstadt https://www.aiphes.tu-darmstadt.de Abstract Concept maps can be used to concisely represent important information and bring structure into large document collections. Therefore, we study a variant of multi- document summarization that produces summaries in the form of concept maps. However, suitable evaluation datasets for this task are currently missing. To close this gap, we present a newly created cor- pus of concept maps that summarize het- erogeneous collections of web documents on educational topics. It was created us- ing a novel crowdsourcing approach that allows us to efficiently determine impor- tant elements in large document collec- tions. We release the corpus along with a baseline system and proposed evaluation protocol to enable further research on this variant of summarization.1 1 Introduction Multi-document summarization (MDS), the trans- formation of a set of documents into a short text containing their most important aspects, is a long- studied problem in NLP. Generated summaries have been shown to support humans dealing with large document collections in information seek- ing tasks (McKeown et al., 2005; Ma˜na-L´opez et al., 2004; Roussinov and Chen, 2001). How- ever, when exploring a set of documents manually, humans rarely write a fully-formulated summary for themselves. Instead, user studies (Chin et al., 2009; Kang et al., 2011) show that they note down important keywords and phrases, try to identify re- lationships between them and organize them ac- cordingly. Therefore, we believe that the study of 1Available at https://github.com/UKPLab/ emnlp2017-cmapsum-corpus Concepts Concept Maps Infor- mation consist of are used to structure Concept Relation Proposition Figure 1: Elements of a concept map. summarization with similarly structured outputs is an important extension of the traditional task. A representation that is more in line with ob- served user behavior is a concept map (Novak and Gowin, 1984), a labeled graph showing con- cepts as nodes and relationships between them as edges (Figure 1). Introduced in 1972 as a teach- ing tool (Novak and Ca˜nas, 2007), concept maps have found many applications in education (Ed- wards and Fraser, 1983; Roy, 2008), for writing assistance (Villalon, 2012) or to structure infor- mation repositories (Briggs et al., 2004; Richard- son and Fox, 2005). For summarization, concept maps make it possible to represent a summary con- cisely and clearly reveal relationships. Moreover, we see a second interesting use case that goes be- yond the capabilities of textual summaries: When concepts and relations are linked to correspond- ing locations in the documents they have been ex- tracted from, the graph can be used to navigate in a document collection, similar to a table of contents. An implementation of this idea has been recently described by Falke and Gurevych (2017). The corresponding task that we propose is concept-map-based MDS, …

Content truncated to 500 words