Crowdsourced Corpus with Entity Salience Annotations

Crowdsourced Corpus with Entity Salience Annotations Milan Dojchinovski1, Dinesh Reddy2, Tomáš Kliegr3, Tomas Vitvar1, Harald Sack2 1 Web Intelligence Research Group Faculty of Information Technology, Czech Technical University in Prague firstname.lastname@fit.cvut.cz Semantic Technologies Group 2 Hasso Plattner Institute, University of Potsdam, Germany firstname.lastname@hpi.de 3 Department of Information and Knowledge Engineering Faculty of Informatics and Statistics, University of Economics, Prague firstname.lastname@vse.cz Abstract In this paper, we present a crowdsourced dataset which adds entity salience (importance) annotations to the Reuters-128 dataset, which is subset of Reuters-21578. The dataset is distributed under a free license and publish in the NLP Interchange Format, which fosters interoperability and re-use. We show the potential of the dataset on the task of learning an entity salience classifier and report on the results from several experiments. Keywords: document aboutness, named entities, entity salience, entity importance, text analysis 1. Introduction Understanding the aboutness of Web documents important for many Web based systems. While in the pastis the aboutness of documents has been primarily formulated as a problem of finding the top-K most relevant keywords in the document (Tomokiyo and Hurst, 2003; Hulth, 2003) it is a recent trend that many NLP systems are moving towards understanding documents in terms of entities. Currently, there are many named entity recognition systems (NER). Nevertheless, in long texts, the list of recognized entities can be very large containing also entities which are not relevant to the document. Identifying a subset of salient entities that play an important role in a story that the document describes can significantly help to better understand the aboutness of the document. To this end, the research community has explored already existing datasets for the entity salience task, e.g. the Microsoft Document Aboutness (MDA) dataset, and the New York Times (NYT) dataset. However, neither dataset provides the underlying document content due to copyright restrictions. Moreover, in these datasets the entity candidates have been generated automatically using proprietary NER systems. To the best of our knowledge, in this paper we present the first publicly available corpus with crowdsourced entity salience annotations. The corpus is created by re-using the NEL Reuters–128 corpus (Röder et al., 2014) and published in the NLP Interchange Format (NIF) ensuring high interoperability and re-use. The corpus is available for download under an open license. To validate and show the potential of the dataset we develop an entity salience learning use case. The use case is based on the crowdsourced dataset and it leverages the local and global information about entities derived from the document itself and the DBpedia knowledge graph. We have conducted several experiments and report the results. The remainder of this paper is organized as follows. Section 2 motivates our work by reviewing the availability of entity salience corpora. Section 3 describes the methodology we followed to develop our entity salience corpus. Section 4 describes the use case showing how this dataset can be used for training an entity salience classifier. It also describes the experiments we run to validate and evaluate the developed …

Content truncated to 500 words