Modeling Content Identification from Document Images

Modeling Content Identification from Document Images Takehiro Nakayama Fuji Xerox Palo Alto Laboratory 3400 HiUview Avenue Palo Alto, CA 94304 USA nakayama~pal.xerox.com Abstract A new technique to locate content-represent- ing words for a given document image using abstract representation of character shapes is described. A character shape code representa- tion defined by the location of a character in a text line has been developed. Character shape code generation avoids the computational expense of conventional optical character rec- ognition (OCR). Because character shape codes are an abstraction of standard character code (e.g., ASCII), the mapping is ambiguous. In this paper, the ambiguity is shown to be practically limited to an acceptable level. It is illustrated that: first, punctuation marks are clearly distinguished from the other charac- ters; second, stop words are generally distin- guishable from other words, because the permutations of character shape codes in func- tion words are characteristically different from those in content words; and third, numerals and acronyms in capital letters are distinguish- able from other words. With these clAssifica- tions, potential content-representing words are identified, and an analysis of their distribution yields their rank. Consequently, introducing character shape codes makes it possible to inexpensively and robustly bridge the gap between electronic documents and hard-copy documents for the purpose of content identifi- cation. 1 Introduction Documents are becoming increasingly available in machine-readable form. As they are stored automati- cally and transferred on networks, many natural lan- guAge processing techniques that identify their content have been developed to assist users with information retrieval and document classification. Conventionally, stored records axe identified by sets of keywords or phrases, known as index terms (Salton, 1991). Although documents are increasingly being com- puter generated, they are still printed on paper for read- ing, dissemination, and markup. As it is believed that paper will remain a comfortable medium for reading and modification (O'Gorman and Kasturi, 1992), devel- opment of content identifiCAtion techniques from a doc- ument image is still important. OCR is often used to convert a document image into machine-readable form, but processing performance is limited by the overhead of OCR (Mori et al., 1992; Nagy, 1992; Rice et al., 1993). Because of the inaccuracy and expense of OCR, we decided to avoid using it. Instead, we have developed a method that first makes generalizations about images of characters, then performs gross classification of the isolated characters and agglomerates these character shape codes into spa- tially isolated (word shape) tokens (Nakayama and Spitz, 1993; Sibun and Spitz, this volume). Generating word shape tokens is inexpensive, fast, and robust. Word shape tokens are a potential alternative to charac- ter coded words when they are used for language deter- mination and part-of-speech tagging (Nakayama and Spitz, 1993; Sibun and Spitz, this volume; Sibun and Farrar, 1994). In this paper, we describe an extension of our approach to content identification. 2 Word shape token generation from image In this section, we introduce word shape tokens and their generation from document images. First, we classify characters …

Content truncated to 500 words