Evaluating Models of Latent Document Semantics in the Presence of OCR Errors

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 240–250, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Evaluating Models of Latent Document Semantics in the Presence of OCR Errors Daniel D. Walker, William B. Lund, and Eric K. Ringger Brigham Young University Provo, Utah, USA danl4@cs.byu.edu, bill lund@byu.edu, ringger@cs.byu.edu Abstract Models of latent document semantics such as the mixture of multinomials model and La- tent Dirichlet Allocation have received sub- stantial attention for their ability to discover topical semantics in large collections of text. In an effort to apply such models to noisy optical character recognition (OCR) text out- put, we endeavor to understand the effect that character-level noise can have on unsu- pervised topic modeling. We show the ef- fects both with document-level topic analy- sis (document clustering) and with word-level topic analysis (LDA) on both synthetic and real-world OCR data. As expected, experi- mental results show that performance declines as word error rates increase. Common tech- niques for alleviating these problems, such as filtering low-frequency words, are successful in enhancing model quality, but exhibit fail- ure trends similar to models trained on unpro- cessed OCR output in the case of LDA. To our knowledge, this study is the first of its kind. 1 Introduction As text data becomes available in massive quanti- ties, it becomes increasingly difficult for human cu- rators to manually catalog and index modern docu- ment collections. To aid in the automation of such tasks, algorithms can be used to create models of the latent semantics present in a given corpus. One example of this type of analysis is document cluster- ing, in which documents are grouped into clusters by topic. Another type of topic analysis attempts to discover finer-grained topics—labeling individual words in a document as belonging to a particular topic. This type of analysis has grown in popular- ity recently as inference on models containing large numbers of latent variables has become feasible. The modern explosion of data includes vast amounts of historical documents, made available by means of Optical Character Recognition (OCR), which can introduce significant numbers of er- rors. Undertakings to produce such data include the Google Books, Internet Archive, and HathiTrust projects. In addition, researchers are having increas- ing levels of success in digitizing hand-written man- uscripts (Bunke, 2003), though error rates remain much higher than for OCR. Due to their nature, these collections often lack helpful meta-data or labels. In the absence of such labels, unsupervised machine learning methods can reveal patterns in the data. Finding good estimates for the parameters of models such as the mixture of multinomials docu- ment model (Walker and Ringger, 2008) and the La- tent Dirichlet Allocation (LDA) model (Blei et al., 2003) requires accurate counts of the occurrences and co-occurrences of words. Depending on the age of a document and the way in which it was created, the OCR process results in text containing many types of noise, including character-level er- rors, which distort these …

Content truncated to 500 words