Parsing the LOB Corpus

PARSING THE LOB CORPUS Carl G. de Marcken MIT AI Laboratory Room 838 545 Technology Square Cambridge, MA 02142 Internet: cgdemarc@ai.mit.edu ABSTRACT This paper 1 presents a rapid and robust pars- ing system currently used to learn from large bodies of unedited text. The system contains a multivalued part-of-speech disambiguator and a novel parser employing bottom-up recogni- tion to find the constituent phrases of larger structures that might be too difficult to ana- lyze. The results of applying the disambiguator and parser to large sections of the Lancaster/ Oslo-Bergen corpus are presented. INTRODUCTION We have implemented and tested a pars- ing system which is rapid and robust enough to apply to large bodies of unedited text. We have used our system to gather data from the Lancaster/Oslo-Bergen (LOB) corpus, generat- ing parses which conform to a version of current Government-Binding theory, and aim to use the system to parse 25 million words of text The system consists of an interface to the LOB corpus, a part of speech disambiguator, and a novel parser. The disambiguator uses multivaluedness to perform, in conjunction with the parser, substantially more accurately than current algorithms. The parser employs bottom- up recognition to create rules which fire top- down, enabling it to rapidly parse the constituent phrases of a larger structure that might itself be difficult to analyze. The complexity of some of the free text in the LOB demands this, and we have not sought to parse sentences completely, but rather to ensure that our parses are accu- rate. The parser output can be modified to con- form to any of a number of linguistic theories. This paper is divided into sections discussing the LOB corpus, statistical disambiguation, the parser, and our results. 1 This paper reports work done at the MIT Artificial Intelligence Laboratory. Support for this research was provided in part by grants from the National Science Foundation (under a Presidential Young Investigator award to Prof. Robert C. Berwick); the Kapor Family Foun- dation; and the Siemens Corporation. THE LOB CORPUS The Lancaster/Oslo-Bergen Corpus is an on- line collection of more than 1,000,000 words of English text taken from a variety of sources, broken up into sentences which are often 50 or more words long. Approximately 40,000 differ- ent words and 50,000 sentences appear in the corpus. We have used the LOB corpus in a standard way to build several statistical tables of part of speech usage. Foremost is a dictionary keying every word found in the corpus to the number of times it is used as a certain part of speech, which a/lows us to compute the probability that a word takes on a given part of speech. In ad- dition, we recorded the number of times each part of speech occurred in the corpus, and built a digram array, listing the number of times one part of speech was followed by another. These numbers can be used to compute the probability of one category preceding another. Some disambiguation schemes require knowing …

Content truncated to 500 words