An Open Corpus for Named Entity Recognition in Historic Newspapers

An Open Corpus for Named Entity Recognition in Historic Newspapers Clemens Neudecker Staatsbibliothek zu Berlin Potsdamer Straße 33, 10785 Berlin clemens.neudecker@sbb.spk-berlin.de Abstract The availability of openly available textual datasets (“corpora”) with highly accurate manual annotations (“gold standard”) of named entities (e.g. persons, locations, organizations, etc.) is crucial in the training and evaluation of named entity recognition systems. Currently there are only few such datasets available on the web, and even less for texts containing historical spelling variation. The production and subsequent release into the public domain of four such datasets with 100 pages each for the languages Dutch, French, German (including Austrian) as part of the Europeana Newspapers project is expected to contribute to the further development and improvement of named entity recognition systems with a focus on historical content. This paper describes how these datasets were produced, what challenges were encountered in their creation and informs about their final quality and availability. Keywords: Named Entity Recognition, Historic Newspapers, Optical Character Recognition 1. Introduction Europeana Newspapers 1 is a collaborative European research and digitization project co-funded by the European Commission that brought together 18 partners from 12 countries to aggregate full text and scanned images of approximately 12 million pages of historic European newspapers and make them available via an online service provided by The European Library2. The entire collection comprises more than 1,000 digitized newspaper titles from 23 libraries across Europe, spanning news articles from four centuries (from 1618 up to 1990) in more than 40 different languages. Based on scans of the newspaper pages, full texts were produced by processing the images with optical character recognition (OCR) using state-of-the-art software tools. A conservative estimate of 2,000 words per full newspaper page leads to a total amount of approximately 25 billion tokens in the overall corpus that can be searched. Figure 1: Example scan of a title page from the French newspaper “Le Temps” 21 http://www.europeana-newspapers.eu/. The European Library: Historic Newspapers, http://www.theeuropeanlibrary.org/tel4/newspapers. Named entity recognition (NER) is the process of automatic extraction of named entities by means of recognition (finding the entities in a given text) and their classification (assigning a type). The application of named entity recognition to the full text collection derived by means of OCR can dramatically improve the usability of the collection, since it allows users to find specific individuals or places in large text collections. It can also be extremely useful for automatic indexing, automatic summarisation, topic detection, trend analysis and numerous data analysis and visualisation applications. Furthermore, it enables additional refinements such as the cross-linking of the newspaper content with other online information resources like authority files and the linked (open) data cloud. This is particularly relevant in the case of newspaper content. An analysis of the web log files of the digital newspaper collection from the National Library of Wales (Gooding, 2014) indicated that up to 9 out of 10 user queries are either person or place names, rather than standard keywords. This is a specific feat of digital newspaper collections, where …

Content truncated to 500 words