Using SGML as a Basis for Data-Intensive NLP

Using SGML as a Basis for Data-Intensive NLP David McKelvie, Chris Brew & Henry Thompson Language Technology Group, Human Communication Research Centre, University of Edinburgh, Edinburgh, Scotland David. McKelvie@ed. ac. uk ~z Chris. Brew@ed. ac. uk & H. Thompson@ed. ac. uk Abstract This paper describes the LT NSL sys- tem (McKelvie et al, 1996), an architec- ture for writing corpus processing tools. This system is then compared with two other systems which address similar is- sues, the GATE system (Cunningham et al, 1995) and the IMS Corpus Workbench (Christ, 1994). In particular we address the advantages and disadvantages of an SGML approach compared with a non-SGML database approach. 1 Introduction The theme of this paper is the design of software and data architectures for natural language process- ing using corpora. Two major issues in corpus-based NLP are: how best to deal with medium to large scale corpora often with complex linguistic annota- tions, and what system architecture best supports the reuse of software components in a modular and interchangeable fashion. In this paper we describe the LT NSL system (McK- elvie et al, 1996), an architecture for writing corpus processing tools, which we have developed in an at- tempt to address these issues. This system is then compared with two other systems which address some of the same issues, the GATE system (Cun- ningham et al, 1995) and the IMS Corpus Work- bench (Christ, 1994). In particular we address the advantages and disadvantages of an SGML approach compared with a non-SGML database approach. Fi- nally, in order to back up our claims about the merits of SGML-based corpus processing, we present a num- ber of case studies of the use of the LT NSL system for corpus preparation and linguistic analysis. 2 The LT NSL system LT NSL is a tool architecture for SGML-based pro- cessing of (primarily) text corpora. It generalises the UNIX pipe architecture, making it possible to use pipelines of general-purpose tools to process an- notated corpora. The original UNIX architecture al- lows the rapid construction of efficient pipelines of conceptually simple processes to carry out relatively complex tasks, but is restricted to a simple model of streams as sequences of bytes, lines or fields. LT NSL lifts this restriction, allowing tools access to streams which are sequences of tree-structured text (a repre- sentation of SGML marked-up text). The use of SGML as an I/0 stream format between programs has the advantage that SGML is a well de- fined standard for representing structured text. Its value is precisely that it closes off the option of a proliferation of ad-hoc notations and the associated software needed to read and write them. The most important reason why we use SGML for all corpus lin- guistic annotation is that it forces us to formally de- scribe the markup we will be using and provides soft- ware for checking that these markup invariants hold in an annotated corpus. In practise this is extremely useful. SGML is human readable, so that interme- …

Content truncated to 500 words