A Language Modelling Approach to Quality Assessment of OCR’ed Historical Text

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5859–5864 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5859 A Language Modelling Approach to Quality Assessment of OCR’ed Historical Text Callum W Booth, Robert Shoemaker, Robert Gaizauskas University of Sheffield, Sheffield, United Kingdom {cwbooth1, r.shoemaker, r.gaizauskas}@sheffield.ac.uk Abstract We hypothesise and evaluate a language model-based approach for scoring the quality of OCR transcriptions in the British Library Newspapers (BLN) corpus parts 1 and 2, to identify the best quality OCR for use in further natural language processing tasks, with a wider view to link individual newspaper reports of crime in nineteenth-century London to the Digital Panopticon—a structured repository of criminal lives. We mitigate the absence of gold standard transcriptions of the BLN corpus by utilising a corpus of genre-adjacent texts that capture the common and legal parlance of nineteenth-century London—the Proceedings of the Old Bailey Online—with a view to rank the BLN transcriptions by their OCR quality. Keywords: ocr, evaluation, language modelling, information extraction from historical text 1. Introduction This work takes place as part of a larger project to de- tect and link entities from nineteenth-century historical crime records in the London press, to the Digital Panop- ticon (www.digitalpanopticon.org, 2022)—a repository of structured criminal ‘life archives’. We target Gale’s British Library Newspapers (BLN) parts 1 and 2 datasets (Gale, 2022) as the primary data source. In order to work with these corpora tractably, we must be able to mitigate two limiting factors—dataset size, and transcription quality. Not all of the 14 million doc- uments in the combined corpus will be relevant to the project, or of a high enough quality for downstream tasks. OCR quality within the BLN corpus is highly variable due to factors including original scan qualities (a prod- uct of microfilm/document quality, ink and paper qual- ity, physical degradation along the spine and areas with high hand contact, topology of the paper during scan, scan resolution, etc. (Holley, 2009)), the machine tran- scription process used to create the OCR (a process which varied over the course of corpus creation1), and modern typeface biases in OCR engines (most engines are trained to handle modern typefaces, and are lack- ing for historical documents (Springmann and Lüdeling, 2017)). We wish to address both the corpus size problem and the OCR quality problem by ranking documents by their transcription quality, and then discarding docu- ments below a quality threshold. Gale’s BLN platform provides an “OCR Confidence" metric that could po- tentially serve as a candidate quality metric, however this would be problematic in this context for a number of reasons. The OCR confidence metric provided by the platform is an aggregation of individual character confidence values pulled directly from the OCR engine 1Personal communication with Gale representatives, Octo- ber 26, 2020 used2. Additionally, character level confidence does not equate to accuracy of word transcription—engine- supplied confidence metrics cannot provide confirma- tion that a character is correct in context—verification at the word level …

Content truncated to 500 words