Acquiring Disambiguation Rules From Text

ACQUIRING DISAMBIGUATION RULES FROM TEXT Donald Hindle AT~T Bell Laboratories 600 Mountain Avenue Murray Hill, NJ 07974-2070 Abstract An effective procedure for automatically acquiring a new set of disambiguation rules for an existing deterministic parser on the basis of tagged text is presented. Performance of the automatically ac- quired rules is much better than the existing hand- written disambiguation rules. The success of the acquired rules depends on using the linguistic in- formation encoded in the parser; enhancements to various components of the parser improves the ac- quired rule set. This work suggests a path toward more robust and comprehensive syntactic analyz- ers. 1 Introduction One of the most serious obstacles to developing parsers to effectively analyze unrestricted English is the difficulty of creating sufllciently comprehen- sive grammars. While it is possible to develop toy grammars for particular theoretically interest- ing problems, the sheer variety of forms in En- glish together with the complexity of interaction that arises in a typical syntactic analyzer makes each enhancement of parser coverage increasingly difficult. There is no question that we are still quite far f~om syntactic analyzers that even begin to adequately model the grammatical variety of English. To go beyond the current generation of hand built grAmrnars for syntactic analysis it will be necessary to develop means of acquiring some of the needed grammatical information from the regularities that appear in large corpora of natu- rally occurring text. This paper describes an implemented training procedure for automatically acquiring symbolic rules for a deterministic parser on the basis of un- restricted textual input. In particular, I describe experiments in automatically acquiring a set of rules for disambiguation of lexical category (part of speech). Performance of the acquired rule set is much better than the set of rules for lexical dis- ambiguation written for the parser by hand over a period of several rules; the error rate is approx- imately half that of the hand written rules. Fur- thermore, the error rate is comparable to recent probabilistic approaches such as Church (1987) and Garside, Leech and Sampson (1987). The current approach has the added advantage that, since the rules acquired depend on the parser's grammar in general, independent improvements in other modules of the parser can lead to improve- ment in the performance of the disambiguation component. 2 Categorial Ambiguity Ambiguity of part of speech is a pervasive char- acteristic of English; more than a third of the word tokens in the million-word "Brown Corpus" of written English (Francis and Kucera 1982) are cate$orially ambiguous. It is possible to construct sentences in which every word is ambiguous, such as the following, (1) Her hand had come to rest on that very book. But even without such contrived exaggeration, ambiguity of lsxical category is not a trivial prob- lem. Nor can part of speech ambiguity be ig- nored in constructing models of natural language processing, since syntactic analysis (as well as higher levels of analysis) depends on correctly dis- ambiguating the lexical category of both …

Content truncated to 500 words