A Novel Machine Learning Based Approach for Post-OCR Error Detection

Proceedings of Recent Advances in Natural Language Processing, pages 1463–1470 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_164 1463 A Novel Machine Learning Based Approach for Post-OCR Error Detection Shafqat Mumtaz Virk Spr˚akbanken Text, Dept. of Swedish University of Gothenburg 405 30 Gothenburg, Sweden virk.shafqat@gmail.com Dana Dann´ells Spr˚akbanken Text, Dept. of Swedish University of Gothenburg 405 30 Gothenburg, Sweden dana.dannells@svenska.gu.se Azam Sheikh Muhammad Chalmers University of Technology 412 96 Gothenburg, Sweden sheikhazam@gmail.com Abstract Post processing is the most conventional ap- proach for correcting errors that are caused by Optical Character Recognition (OCR) sys- tems. Two steps are usually taken to correct OCR errors: detection and corrections. For the first task, supervised machine learning meth- ods have shown state-of-the-art performances. Previously proposed approaches have focused most prominently on combining lexical, con- textual and statistical features for detecting er- rors. In this study, we report a novel system to error detection which is based merely on the n-gram counts of a candidate token. In addi- tion to being simple and computationally less expensive, our proposed system beats previous systems reported in the ICDAR2019 competi- tion on OCR-error detection with notable mar- gins. We achieved state-of-the-art F1-scores for eight out of the ten involved European lan- guages. The maximum improvement is for Spanish which improved from 0.69 to 0.90, and the minimum for Polish from 0.82 to 0.84. 1 Introduction Post processing is the most conventional approach for correcting errors that are caused by Optical Character Recognition (OCR) systems. Tradition- ally, the task is divided into two subtasks: (1) Error detection to classify words1 as either erroneous or valid, and (2) Error correction to find suitable can- didates to correct the erroneous words (Kolak and Resnik, 2005; Kissos and Dershowitz, 2016; Mei et al., 2016). A large body of work has proven the success of statistical and supervised machine learn- ing methods for both subtasks (Afli et al., 2016; 1We write ‘word’ although, in practice, it actually refers to any token in our data – not necessarily a lexical word. We will use the term ’token’ and ’word’ alternatively throughout the paper. Schulz and Kuhn, 2017; Nguyen et al., 2018; Am- rhein and Clematide, 2018; Nguyen et al., 2019). Machine learning methods largely rely on fea- ture engineering for their performances, particu- larly in supervised settings. Feature engineering in- volves exploring various features and feature com- binations that best characterise the data. However, for post-OCR error detection, finding a suitable set of features is challenging because of the diver- sity of OCR errors (Amrhein and Clematide, 2018). This has been demonstrated in previous work and more recently in the ICDAR competitions (Chiron et al., 2017; Rigaud et al., 2019) where various features have been explored with varying success rates (more details in Section 2). To address this challenge we propose a novel approach to the error detection task. Instead of examining more features we focus merely on a single feature, namely the n-gram counts of the candidate token. Our approach is inspired by dic- tionary lookup approaches, …

Content truncated to 500 words