Language Determination: Natural Language Processing from Scanned Document Images

Language Determination: Natural Language Processing from Scanned Document Images Penelope Sibun & A. Lawrence Spitz Fuji Xerox Palo Alto Laboratory 3400 HiUview Avenue Palo Alto, CA 94304 USA sibun@ pal. xerox.corn Abstract Many documents are available to a computer only as images from paper. However, most nat- ural language processing systems expect their input as character-coded text, which may be difficult or expensive to extract accurately from the page. We describe a method for con- verting a document image into character shape codes and word shape tokens. We believe that this representation, which is both cheap and robust, is sufficient for many NLP tasks. In this paper, we show that the representation is suffi- cient for determining which of 23 languages the document is written in, using only a small number of features, with greater than 90% accuracy overall. 1 Introduction Computational linguists work with texts. Computational lin!mistic applications range from natural language understanding to information retrieval to machine trans- lation. Such systems usually assume the language of the text that is being processed. However, as corpora become larger and more diverse this assumption becomes less warranted. Attention is now turning to the issue of determining the language or languages of a text before further processing is done. Several sources of information for language determination have been tried: short words (Kulikowski 1991, Ingle 1976); n-grams of words (Batchelder 1992); n-grams of characters (Cavner & Trenkle 1994); diacritics and special characters (Beesley 1988, Newman 1987); syllable characteristics (Mustonen 1965); morphology and syntax (Ziegler 1991). F~ch of these approaches is prGmising although none is completely accurate. More fundamentally, many rely on relatively large amounts of text data and all rely on data in the form of character codes (e.g., ASCID. In today's world of text-based information, how- ever, not all sources of text will be character coded. Many documents such as incoming faxes, patent appli- cations, and office memos are only accessible on paper. Processes such as Optical Character Recognition (OCR) have been developed for mapping paper documents into character-coded text. However, for applications like OCR, it is desirable to know the language a document is in before trying to decode its characters. There appears to be a fundamental Catch-22: natural language processing systems want to be able to work automatically with arbitrary documents, many of which may be available only on paper, and in the process, they minimally need to know which lan- guage or languages are present. The algorithms cited above can determine a document's language, but they require a character-coded representation of the text. OCR can produce such a representation, but OCR does not work well unless the language(s) of the document are known. So how can the language of a paper docu- ment be determined? We have developed a method which reliably deter- mines the language or lan£xlages of a document image. In this paper, we discuss Roman-alphabet languages such as English, Polish, and Swahili; see Spitz (1994) for a discussion of the determination of Asian-script lan- guages. …

Content truncated to 500 words