Development and Evaluation of a Broad-Coverage Probabilistic Grammar of English-Language Computer Manuals

Development and Evaluation of a Broad-Coverage Probabilistic Grammar of English-Language Computer Manuals Ezra Black John Lafferty Salim Roukos <black I j laff ] roukos>*watson, ibm. tom IBM Thomas J. Watson Research Center, P.O. Box 704, Yorktown Heights, New York 10598 ABSTRACT We present an approach to grammar development where the task is decomposed into two separate subtasks. The first task is hnguistic, with the goal of producing a set of rules that have a large coverage (in the sense that the correct parse is among the proposed parses) on a bhnd test set of sentences. The second task is statistical, with the goal of developing a model of the grammar which assigns maximum probability for the correct parse. We give parsing results on text from computer manuals. 1. Introduction Many language understanding systems and machine translation systems rely on a parser of English as the first step in processing an input sentence. The general impres- sion may be that parsers with broad coverage of English are readily available. In an effort to gauge the state of the art in parsing, the authors conducted an experiment in Summer 1990 in which 35 sentences, all of length 13 words or less, were selected randomly from a several-million- word corpus of Associated Press news wire. The sentences were parsed by four of the major large-coverage parsers for general English. 1 Each of the authors, working sep- arately, scored 140 parses for correctness of constituent boundaries, constituent labels, and part-of-speech labels. All that was required of parses was accuracy in delim- iting and identifying obvious constituents such as noun phrases, prepositional phrases, and clauses, along with at least rough correctness in assigning part-of-speech labels, e.g. a noun could not be labelled as a verb. The tallies of each evaluator were compared, and were identical or very close in all cases. The best-performing parser was correct for 60% of the sentences and the the remaining parsers were below 40%. More recently, in early 1992, the cre- ator of another well-known system performed self-scoring on a similar task and reported 30% of input sentences as having been correctly parsed. On the basis of the pre- ceeding evidence it seems that the current state of the t At least one of the parties involved insisted that no perfor- mance results be made public. Such reticence is widespread and understandable. However, it is nonetheless important that perfor- mance norms be established for the field. Some progress has been made in this direction [3, 4]. art is far from being able to produce a robust parser of general English. In order to break through this bottleneck and begin making steady and quantifiable progress toward the goal of developing a highly accurate parser for general En- glish, organization of the grammar-development process along scientific lines and the introduction of stochastic modelling techniques are necessary, in our view. We have initiated a research program on these principles, which we describe in what follows. An account of our overall method of attacking …

Content truncated to 500 words