OCR Post Correction for Endangered Language Texts

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5931–5942, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5931 OCR Post Correction for Endangered Language Texts Shruti Rijhwani,1 Antonios Anastasopoulos,2, † Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University 2Department of Computer Science, George Mason University {srijhwan,gneubig}@cs.cmu.edu, antonis@gmu.edu Abstract There is little to no data available to build nat- ural language processing models for most en- dangered languages. However, textual data in these languages often exists in formats that are not machine-readable, such as paper books and scanned images. In this work, we address the task of extracting text from these resources. We create a benchmark dataset of transcrip- tions for scanned books in three critically en- dangered languages and present a systematic analysis of how general-purpose OCR tools are not robust to the data-scarce setting of en- dangered languages. We develop an OCR post- correction method tailored to ease training in this data-scarce setting, reducing the recogni- tion error rate by 34% on average across the three languages.1 1 Introduction Natural language processing (NLP) systems exist for a small fraction of the world’s over 6,000 liv- ing languages, the primary reason being the lack of resources required to train and evaluate models. Technological advances are concentrated on lan- guages that have readily available data, and most other languages are left behind (Joshi et al., 2020). This is particularly notable in the case of endan- gered languages, i.e., languages that are in danger of becoming extinct due to dwindling numbers of native speakers and the younger generations shift- ing to using other languages. For most endangered languages, finding any data at all is challenging. In many cases, natural language text in these languages does exist. However, it is locked away in formats that are not machine-readable — pa- per books, scanned images, and unstructured web pages. These include books from local publishing †: Work done at Carnegie Mellon University. 1Code and data are available at https://shrutirij. github.io/ocr-el/. (a) Ainu (left) – Japanese (right) (b) Griko (top) – Italian (bottom) (c) Yakkha (top) – Nepali (middle) – English (bottom) (d) Handwritten Shangaji – typed English glosses Text 31: cashew nuts Amina Sharaama explains how to make a sauce of green cashew nuts. R April 2007. notebooks: p. 1230 31.1 nxúuzi wa náńtiíkwa mu-xuzi o-a nantikwa 3-sauce 3-Conn 1a.cashew A sauce of green cashew nuts. 31.2 mí kittóonxipilíkáari nhaáno wírá nxúzí wa náńtiíkwa mi ki-ttoo-mu-xipilikari-a mu-hano o-ir-a mu-xuzi o-a na 1sg.Subst 1sg-Prog-1-explain-Fi 3-white.woman 15-say-Inf 3-sauce 3-C I am explaining the white woman about a sauce of green cashew nuts. 31.3 nańtiíkwa khaaju* khaáju t'íiniyaá nańtiíkwa nantikwa khaju khaju khaju ti e-ni-iy-a_yo nantikwa 1a.cashew 9;cashew 9.cashew Cop 9-Pres-be-Fi_Rel 1a.cashew Green cashew nuts are cashew nuts, cashew nuts is what are green cas 31.4 khajú yáaw'etíile eri étteétthe esikhoomaléeni khaju e-awe entile e-ri e-ttetthe e-si-khoomal-eni 9.cashew 9-Poss.1 9.Demiii 9-be 9-unripe 9-Neg-be.fully.grown-PSit these very cashew nuts are unripe, they are not fully grown yet 31.5 masí esikhoomaléeni n'eesiyéeni étteétthe masi e-si-khoomal-eni …

Content truncated to 500 words