Building OCR/NER Test Collections

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4639–4646 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4639 Building OCR/NER Test Collections Dawn Lawrie, James Mayfield, David Etter HLTCOE at Johns Hopkins University Baltimore, MD 21211 {lawrie,mayfield}@jhu.edu, etterd@gmail.com Abstract Named entity recognition (NER) identifies spans of text that contain names. Many researchers have reported the results of NER on text created through optical character recognition (OCR) over the past two decades. Unfortunately, the test collections that support this research are annotated with named entities after optical character recognition (OCR) has been run. This means that the collection must be re-annotated if the OCR output changes. Instead, by tying annotations to character locations on the page, a collection can be built that supports OCR and NER research without requiring re-annotation when either improves. This means that named entities are annotated on the transcribed text. The transcribed text is all that is needed to evaluate the performance of OCR. For NER evaluation, the tagged OCR output is aligned to the transcription, and modified versions of each are created and scored. This paper presents a methodology for building such a test collection and releases a collection of Chinese OCR-NER data constructed using the methodology. The paper provides performance baselines for current OCR and NER systems applied to this new collection. Keywords: OCR and NER test collection, Chinese named entities, OCR evaluation, NER evaluation 1. Introduction Named entity recognition (NER) is the automatic recogni- tion of spans of text as name mentions, and the categoriza- tion of those spans into a predefined set of types such as people, places, organizations, software, etc. Most NER re- search is performed using text that is created in digital form, such as newswire, blog posts, etc. When the text is derived from another medium, such as speech or images, it can be degraded in ways not usually seen in digital text. This pa- per presents a collection that can help answer the questions “how can one evaluate NER when the input is corrupted?” and “what is the impact of non-digital source medium on NER scores?” The standard pipeline for performing NER on text images begins with layout analysis and optical character recogni- tion (OCR). If the image is a page from a newspaper, it might include multiple articles, each with its own headline, perhaps laid out over multiple columns. Finding names on the page starts by recognizing characters and their locations on the page. Next, a reading order for those characters is established. The text can then be tokenized and split into sentences. Finally, an NER system labels the tokens by type. Until these inference steps can be handled jointly, a test collection that facilitates both optical character recog- nition (OCR) and NER research must account for the fact that OCR operates at the character level, whereas NER sys- tems generally assume that they are processing tokenized text one sentence at a time. Studying the performance of NER …

Content truncated to 500 words