WTF-LOD - A New Resource for Large-Scale NER Evaluation

WTF-LOD – A New Resource for Large-Scale NER Evaluation Lubomir Otrusina and Pavel Smrz Brno University of Technology, Faculty of Information Technology, IT4Innovations Centre of Excellence Bozetechova 2, 612 66 Brno, Czech Republic {iotrusina,smrz}@fit.vutbr.cz Abstract This paper introduces the Web TextFull linkage to Linked Open Data (WTF-LOD) dataset intended for large-scale evaluation of named entity recognition (NER) systems. First, we present the process of collecting data from the largest publically-available textual corpora, including Wikipedia dumps, monthly runs of the CommonCrawl, and ClueWeb09/12. We discuss similarities and differences of related initiatives such as WikiLinks and WikiReverse. Our work primarily focuses on links from “textfull” documents (links surrounded by a text that provides a useful context for entity linking), de-duplication of the data and advanced cleaning procedures. Presented statistics demonstrate that the collected data forms one of the largest available resource of its kind. They also prove suitability of the result for complex NER evaluation campaigns, including an analysis of the most ambiguous name mentions appearing in the data. Keywords: named entity evaluation, linked open data, CommonCrawl, ClueWeb, Wikipedia 1. Introduction The need to bridge the semantic gap between the semi- structured “web of documents” and the structured “web of knowledge” (Buitelaar and Cimiano, 2008) has led to the development of various semantic enrichment systems in recent years. Named entity (NE) recognition and linking present a key component of semantic enrichment. Tools such as DBpedia Spotlight (Daiber et al., 2013), Illinois Wikifier (Ratinov et al., 2011), or AIDA (Hoffart et al., 2011) enable annotating mentions of NEs in a plain text and “anchoring” the annotations in linked open data resources (most frequently in DBpedia/Wikipedia). Various evaluation campaigns have also appeared that com- pare quality of the NE annotation on collected datasets. Ini- tiatives such as NIST TAC KBP1 – Knowledge Base Pop- ulation – Entity Discovery and Linking Track (Ji et al., ), NEEL2 – Named Entity rEcognition and Linking Chal- lenge on Microposts (Rizzo et al., 2015), or ERD3 – En- tity Recognition and Disambiguation Challenge (Carmel et al., 2014) rank participating systems based on their over- all performance on collections of specific textual fragments (selected sentences, tweets. . .) that had been manually an- notated. As the manual dataset preparation is tedious, the provided training and test data is limited to few thousands of entity mentions. Developers of NER tools can measure improvements in annotation quality w.r.t. a particular avail- able dataset or they can use specific benchmarking frame- works such as NERD (Rizzo and Troncy, 2011) or Ger- bil (Cornolti et al., 2013), embracing several datasets. Since the manual creation of evaluation datasets is expen- sive and time-consuming, researchers started to think how to collect usable data without the burden of additional hu- man labour. Internal links of Wikipedia can provide an apt resource and datasets such as TagMe4 Wiki-Disamb30 and 1http://nlp.cs.rpi.edu/kbp/ 2http://www.lancaster.ac.uk/ Microposts2015 3http://web-ngram.research.microsoft.com/ ERD2014/ 4http://acube.di.unipi.it/tagme-dataset/ Wiki-Annot30 explore this potential. Although the data is rich (2 million mentions), it is suitable as a training dataset rather than a …

Content truncated to 500 words