OCR Post-Correction Evaluation of Early Dutch Books Online - Revisited

OCR Post-Correction Evaluation of Early Dutch Books Online – Revisited Martin Reynaert TiCC, CLST Tilburg University, Radboud University Nijmegen reynaert@uvt.nl Abstract We present further work on evaluation of the fully automatic post-correction of Early Dutch Books Online, a collection of 10,333 18th century books. In prior work we evaluated the new implementation of Text-Induced Corpus Clean-up (TICCL) on the basis of a single book Gold Standard derived from this collection. In the current paper we revisit the same collection on the basis of a sizeable 1020 item random sample of OCR post-corrected strings from the full collection. Both evaluations have their own stories to tell and lessons to teach. Keywords: OCR Post-Correction, Evaluation, Digitized Books, Dutch, Diachronic 1. Introduction In Reynaert (2014b) we announced the pending availabil- ity of a fully-automatic and unsupervised multilingual OCR post-correction system fit for diachronic work1. We also presented this reimplemented Text-Induced Corpus Clean- up system or TICCL applied to the post-correction of a col- lection of over 10,000, mainly late 18th century, diachronic Dutch books as well as an extended evaluation based on the Gold Standard for a single book from 1789 (known as DPO35). While we stand fully by this evaluation, it can be expected that the text of a single volume – a history book specifi- cally directed at children – does not cover the full gamut of phenomena that may occur in the larger collection. We therefore now strive to find a way of charting these phenom- ena and how our system manages to deal with them on the basis of a totally different, but necessarily still limited, sam- ple. What one needs to come to grips with in work as this, is the sheer size of the search space we are dealing with. A list of over 800 million pairs of word variants paired to pos- sible Correction Candidates or CCs is beyond any human’s comprehension. We are quite incapable of even beginning to evaluate this full list. All we can possibly do is query it for a limited subselection and try to comprehend what this limited sample may tell us. This is exactly what we try to do in this paper. In Section 2. we present a new evaluation set randomly sampled from TICCL’s output on the 10K Early Dutch Books Online corpus and briefly discuss how we verified and annotated it. We next in Section 3. derive evalua- tion scores and further sample statistics and analyse them in some depth. The discussion in Section 4. deals with costs and benefits of the two evaluation methods, discusses spe- cific OCR phenomena and finally offers an informed and prioritized list of possible extensions to our own – and pos- sibly other – OCR post-correction systems. 1Available in the CLARIN infrastructure at http:// ticclops.clarin.inl.nl and with a new interface at http://philostei.clarin.inl.nl 2. A new Gold Standard 2.1. TICCL briefly recapitulated In order to comprehend the following it is doubtless nec- essary to be aware that TICCL2 derives what amounts to a …

Content truncated to 500 words