Using SMT for OCR Error Correction of Historical Texts

Using SMT for OCR Error Correction of Historical Texts Haithem Afli, Zhengwei Qiu, Andy Way and Páraic Sheridan ADAPT Centre School of Computing Dublin City University Dublin, Ireland {hafli, zqui, away, psheridan}@computing.dcu.ie Abstract A trend to digitize historical paper-based archives has emerged in recent years, with the advent of digital optical scanners. A lot of paper-based books, textbooks, magazines, articles, and documents are being transformed into electronic versions that can be manipulated by a computer. For this purpose, Optical Character Recognition (OCR) systems have been developed to transform scanned digital text into editable computer text. However, different kinds of errors in the OCR system output text can be found, but Automatic Error Correction tools can help in performing the quality of electronic texts by cleaning and removing noises. In this paper, we perform a qualitative and quantitative comparison of several error-correction techniques for historical French documents. Experimentation shows that our Machine Translation for Error Correction method is superior to other Language Modelling correction techniques, with nearly 13% relative improvement compared to the initial baseline. Keywords: OCR error correction, statistical machine translation, historical documents. 1. Introduction Optical Character Recognition (OCR) systems transform a document image into character-code text. In this process, a document image is segmented into character images in the linear reading order using image-analysis heuristics. An au- tomatic classifier is then applied to determine the character code that most likely corresponds to each character image. However, such systems are still imperfect. There are of- ten mistakes in the scanned texts as the OCR system occa- sionally misrecognizes letters and falsely identifies scanned text, leading to misspellings and linguistic errors in the out- put text (Niklas, 2010). In this context, it is appropriate to investigate how we can automatically correct such OCR outputs. This paper explores the effectiveness of OCR out- put error correction. The correction uses an improved noisy channel model, language modelling and statistical machine translation (SMT) to correct OCR errors. The remainder of the paper is organized as follows: Section 2 provides back- ground information on OCR errors; Section 3 presents re- lated work and our error-correction methodology; Section 4 discusses the results; Section 5 concludes and provides directions for further investigation. 2. OCR errors Optical Character Recognition of historical French docu- ments presents several challenges, including: • Old-style language: some old letters can be used in these kinds of texts such as the "long s" in old French writing which is often confused with the lower-cased " f " in various Roman typefaces and in blackletter. In the example of Figure 1, "belle femme surpasse" can be recognised by automatic systems as "belle femme furpaffe". • Punctuation errors: especially when we have a poor scanning quality (like in Figure 2), punctuation char- acter misrecognition can cause commas or full stops to often occur in the wrong positions. Figure 1: Example of a French sentence in old-style lan- guage using a "long s" character. • Character format: variations in font can also prevent accurate character recognition which …

Content truncated to 500 words