A Workbench for Finding Structure in Texts

A Workbench for Finding Structure in Texts Andrei Mikheev HCRC, Language Technology Group, University of Edinburgh, 2 Buccleuch Place, Edinburgh EH8 9LW, UK. Andrei. Mikheev@ed. ac. uk Steven Finch Thomson Technical Labs, 1375 Piccard Drive, Suite 250, Rockville Maryland, 20850 sfinch@thomtech, com Abstract In this paper we report on a set of compu- tational tools with (n)SGML pipeline data flow for uncovering internal structure in natural language texts. The main idea be- hind the workbench is the independence of the text representation and text analysis phases. At the representation phase the text is converted from a sequence of char- acters to features of interest by means of the annotation tools. At the analysis phase those features are used by statistics gath- ering and inference tools for finding signifi- cant correlations in the texts. The analysis tools are independent of particular assump- tions about the nature of the feature-set and work on the abstract level of feature- elements represented as SGML items. 1 Introduction There is increasing agreement that progress in vari- ous areas of language engineering needs large collec- tions of unconstrained language material. Such cor- pora are emerging and are proving to be important research tools in areas such as lexicography, text un- derstanding and information extraction, spoken lan- guage understanding, the evaluation of parsers, the construction of large-scale lexica, etc. The key idea of corpus oriented language analy- sis is to collect frequencies of "interesting" events and then run statistical inferences on the basis of those frequencies. For instance, one might be inter- ested in frequencies of co-occurences of a word with other words and phrases (collocations) (Smadja, 1993), or one might be interested in inducing word- classes from the text by collecting frequencies of the left and right context words for a word in focus (Finch&Chater, 1993). Thus, the building blocks of the "interesting" events might be words, their morpho-syntactic properties (e.g. part-of-speech, suffix, etc.), phrases or their sub-phrases (e.g. head- noun of a noun group), etc. The "interesting" events usually also specify the relation between those build- ing blocks such as "the two words should occur next to each other or in the same sentence". In this paper we describe a workbench for uncovering that kind of internal structure in natural language texts. 2 Data Level Integration The underlying idea behind our workbench is data level integration of abstract data processing tools by means of structured streams. The idea of using an open set of modular tools with stream input/output (IO) is akin to the philosophy behind UNIX. This allows for localization of specific data processing or manipulation tasks so we can use different combina- tions of the same tools in a pipeline for fulfilling dif- ferent tasks. Our architecture, however, imposes an additional constraint on the IO streams: they should have a common syntactic format which is realized as SGML markup (Goldfarb, 1990). A detailed compar- ison of this SGML-oriented architecture with more traditional data-base oriented architectures can be found in (McKelvie et al., 1997). …

Content truncated to 500 words