OCR Post-Processing for Low Density Languages

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 867–874, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics OCR Post-Processing for Low Density Languages Okan Kolak Computer Science and UMIACS University of Maryland College Park, MD 20742 okan@umiacs.umd.edu Philip Resnik Linguistics and UMIACS University of Maryland College Park, MD 20742 resnik@umiacs.umd.edu Abstract We present a lexicon-free post-processing method for optical character recognition (OCR), implemented using weighted fi- nite state machines. We evaluate the technique in a number of scenarios rele- vant for natural language processing, in- cluding creation of new OCR capabilities for low density languages, improvement of OCR performance for a native com- mercial system, acquisition of knowledge from a foreign-language dictionary, cre- ation of a parallel text, and machine trans- lation from OCR output. 1 Introduction The importance of rapidly retargeting existing natu- ral language processing (NLP) technologies to new languages is widely accepted (Oard, 2003). Statisti- cal NLP models have a distinct advantage over rule based approaches to achieve this goal, as they re- quire far less manual labor; however, training statis- tical NLP methods requires on-line text, which can be hard to find for so-called “low density” languages — that is, languages where few on-line resources ex- ist. In addition, for many languages of interest input data are available mostly in printed form, and must be converted to electronic form prior to processing. Optical character recognition (OCR) is often the only feasible method to perform this conversion, owing to its speed and cost-effectiveness. Unfor- tunately, the performance of OCR systems is far from perfect and recognition errors significantly de- grade the performance of NLP applications. This is true both in resource acquisition, such as automated bilingual lexicon generation (Kolak et al., 2003), and for end-user applications such as rapid machine translation (MT) in the battlefield for document fil- tering (Voss and Ess-Dykema, 2000). Moreover, for low density languages, there simply may not be an OCR system available. In this paper, we demonstrate that via statistical post-processing of existing systems, it is possible to achieve reasonable recognition accuracy for low density languages altogether lacking an OCR sys- tem, to significantly improve on the performance of a trainable commercial OCR system, and even to improve significantly on a native commercial OCR system.1 By taking a post-processing approach, we require minimal assumptions about the OCR system used as a starting point. The proper role of our post-processing approach depends on the language. For languages with little commercial potential for OCR, it may well provide the most practical path for language-specific OCR development, given the expensive and time consum- ing nature of OCR development for new languages and the “black box” nature of virtually all state-of- the-art OCR systems. For languages where native OCR development may take place, it is a fast, prac- tical method that allows entry into a new language until native OCR development catches up. For these, and also for languages where native systems exist, 1Currently we assume the …

Content truncated to 500 words