Evaluation of HTR models without Ground Truth Material

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4395–4404 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4395 Evaluation of HTR models without Ground Truth Material Phillip Benjamin Str¨obel1, Simon Clematide1, Martin Volk1, Raphael Schwitter2, Tobias Hodel3, David Schoch3 1Department of Computational Linguistics, 2Department of Classics, 3Walter Benjamin Kolleg 1,2University of Zurich, 3University of Bern {pstroebel, siclemat, volk}@cl.uzh.ch raphael.schwitter@sglp.uzh.ch {tobias.hodel, david.schoch}@unibe.ch Abstract The evaluation of Handwritten Text Recognition (HTR) models during their development is straightforward: because HTR is a supervised problem, the usual data split into training, validation, and test data sets allows the evaluation of models in terms of accuracy or error rates. However, the evaluation process becomes tricky as soon as we switch from development to application. A compilation of a new (and forcibly smaller) ground truth (GT) from a sample of the data that we want to apply the model on and the subsequent evaluation of models thereon only provides hints about the quality of the recognised text, as do confidence scores (if available) the models return. Moreover, if we have several models at hand, we face a model selection problem since we want to obtain the best possible result during the application phase. This calls for GT-free metrics to select the best model, which is why we (re-)introduce and compare different metrics, from simple, lexicon-based to more elaborate ones using standard language models and masked language models (MLM). We show that MLM-based evaluation can compete with lexicon-based methods, with the advantage that large and multilingual transformers are readily available, thus making compiling lexical resources for other metrics superfluous. Keywords: handwritten text recognition, digital humanities, evaluation, ground truth data, resources, model selection 1. Introduction Optical Character Recognition (OCR) has become a well-established technique for digitising historical printed collections in libraries and archives. At the same time, Handwritten Text Recognition (HTR) is also increasingly finding its way into these institutions (Ter- ras, in press). The digitisation efforts of libraries and archives simplify access to sought-after documents for researchers from various disciplines and allow them to pursue research questions they could either not or only very cumbersomely have answered without digi- tal copies of the sources. Still, we can only consider OCR and HTR to be “re- search facilitators” as long as they perform within rea- sonable accuracy ranges. Several studies have shown that inaccuracies in OCRed documents harm informa- tion retrieval and text mining techniques like named en- tity recognition and linking, topic modelling, and lan- guage modelling (Alex and Burns, 2014; Chiron et al., 2017; van Strien et al., 2020; Hamdi et al., 2020; Pontes et al., 2019; Hill and Hengchen, 2019). But what do we mean by “reasonable accuracy ranges”? Holley (2009) shared the experiences of the National Library of Australia Newspaper Digitisation Program, which found that OCR contractors and li- braries usually agree that “good” OCR means > 98%, “average” OCR between 90% and 98%, and “poor” OCR any score < 90% character accuracy, respec- …

Content truncated to 500 words