A flexible distributed architecture for NLP system development and use

A flexible distributed architecture for NLP system development and use Freddy Y. Y. Choi Artificial Intelligence Group University of Manchester Manchester, U.K. choif@cs.man.ac.uk Abstract We describe a distributed, modular architecture for platform independent natural language sys- tems. It features automatic interface genera- tion and self-organization. Adaptive (and non- adaptive) voting mechanisms are used for inte- grating discrete modules. The architecture is suitable for rapid prototyping and product de- livery. 1 Introduction This article describes TEA 1, a flexible architec- ture for developing and delivering platform in- dependent text engineering (TE) systems. TEA provides a generalized framework for organizing and applying reusable TE components (e.g. to- kenizer, stemmer). Thus, developers are able to focus on problem solving rather than imple- mentation. For product delivery, the end user receives an exact copy of the developer's edition. The visibility of configurable options (different levels of detail) is adjustable along a simple gra- dient via the automatically generated user inter- face (Edwards, Forthcoming). Our target application is telegraphic text compression (Choi (1999b); of Roelofs (Forth- coming); Grefenstette (1998)). We aim to im- prove the efficiency of screen readers for the visually disabled by removing uninformative words (e.g. determiners) in text documents. This produces a stream of topic cues for rapid skimming. The information value of each word is to be estimated based on an unusually wide range of linguistic information. TEA was designed to be a development en- vironment for this work. However, the target application has led us to produce an interesting tTEA is an acronym for Text Engineering Architec- ture. architecture and techniques that are more gen- erally applicable, and it is these which we will focus on in this paper. 2 Architecture I System input and output I I L I I Plug*ins Shared knowledge System control s~ructure Figure 1: An overview of the TEA system framework. The central component of TEA is a frame- based data model (F) (see Fig.2). In this model, a document is a list of frames (Rich and Knight, 1991) for recording the properties about each token in the text (example in Fig.2). A typical TE system converts a document into F with an input plug-in. The information required at the output determines the set of process plug-ins to activate. These use the information in F to add annotations to F. Their dependencies are auto- matically resolved by TEA. System behavior is controlled by adjusting the configurable param- eters. Frame 1: (:token An :pos art :begin_s 1) Frame 2: (:token example :pos n) Frame 3: (:token sentence :pos n) Frame 4: (:token . :pos punc :end_s 1) Figure 2: "An example sentence." in a frame- based data model 615 This type of architecture has been imple- mented, classically, as a 'blackboard' system such as Hearsay-II (Erman, 1980), where inter- module communication takes place through a shared knowledge structure; or as a 'message- passing' system where the modules communi- cate directly. Our architecture is similar to blackboard systems. However, the purpose of F (the shared knowledge structure …

Content truncated to 500 words