An Annotated Corpus and Method for Analysis of Ad-Hoc Structures Embedded in Text

An Annotated Corpus and Method for Analysis of Ad-Hoc Structures Embedded in Text Eric Yeh, John Niekrasz, Dayne Freitag, Richard Rohwer SRI International, yeh@ai.sri.com,niekrasz@ai.sri.com,freitag@ai.sri.com,rohwer@ai.sri.com Abstract We describe a method for identifying and performing functional analysis of structured regions that are embedded in natural language documents, such as tables or key-value lists. Such regions often encode information according to ad hoc schemas and avail themselves of visual cues in place of natural language grammar, presenting problems for standard information extraction algorithms. Unlike previous work in table extraction, which assumes a relatively noiseless two-dimensional layout, our aim is to accommodate a wide variety of naturally occurring structure types. Our approach has three main parts. First, we collect and annotate a a diverse sample of “naturally” occurring structures from several sources. Second, we use probabilistic text segmentation techniques, featurized by skip bigrams over spatial and token category cues, to automatically identify contiguous regions of structured text that share a common schema. Finally, we identify the records and fields within each structured region using a combination of distributional similarity and sequence alignment methods, guided by minimal supervision in the form of a single annotated record. We evaluate the last two components individually, and conclude with a discussion of further work. Keywords: table recognition, semistructured data, information extraction 1. Introduction We address the problem of information extraction from structured regions embedded in unstructured documents. Whereas information extraction systems are generally de- signed to process sentential text, many important “unstruc- tured” genres draw freely on a variety of structured idioms to communicate important information concisely, such as headings, lists, and tables. In general, a complete recovery of the information communicated in such genres requires that these structures be recognized and processed. Although the extensive literature on table processing offers a number of algorithmic recipes, tables are only one im- portant structuring idiom. Moreover, most such approaches make fairly strong assumptions about their inputs, and are therefore of limited general use. We are interested here in the general problem of embedded structure (in all its forms) and are in search of methods that can accommodate as wide a range of structuring phenomena as possible. Consider the following excerpt from a law enforce- ment press release (we have changed names to preserve anonymity): The six defendants arrested in the first indictment are charged with.... They are: Don Juan Williams, a/k/a "Flav," "Flava Flav," and "Flay,"age 32, of Laurel, Maryland; Andre T. Dupont, a/k/a "Dre," age 37, of Riverdale; The indictment alleges that Williams purchased large quantities .... This is an example of a form that might be called a “loosely formatted two-dimensional sequence.” Although it exhibits something of a tabular structure, its inter-cell and inter-row delimiters obey no positional regularity. This example also exemplifies the typical centrality of information communi- cated in structured idioms; often, such information repre- sents the primary payload of a document. Our target use case is that of a language engineer having to survey and develop an extraction strategy for a body of documents. …

Content truncated to 500 words