Tagging accurately - Don’t guess if you know

Tagging accurately- Don't guess if you know Pasi Tapanainen Rank Xerox Research Centre Grenoble Laboratory 6, chemin de Maupertuis 38240 Meylan, France Pasi.Tapanainen@xerox.fr Atro Voutilainen Research Unit for Computational Linguistics University of Helsinki P.O. Box 4 00014 University of Helsinki, Finland Atro.Voutilainen@helsinki.fi Abstract We discuss combining knowledge-based (or rule-based) and statistical part-of-speech taggers. We use two mature taggers, ENGCG and Xerox Tagger, to indepen- dently tag the same text and combine the results to produce a fully disambiguated text. In a 27000 word test sample taken from a previously unseen corpus we achieve 98.5 % accuracy. This paper presents the data in detail. We describe the problems we encountered in the course of combining the two taggers and discuss the problem of evaluating taggers. 1 Introduction This paper combines knowledge-based and statisti- cal methods for part-of-speech disambiguation, tak- ing advantage of the best features of both ap- proaches. The resulting output is fully and accu- rately disambiguated. We demonstrate a system that accurately resolves most part-of-speech ambiguities by means of syntac- tic rules and employs a stochastic tagger to elimi- nate the remaining ambiguity. The overall results are clearly superior to the reported results for state- of-the-art stochastic systems. The input to our part-of-speech disambiguator consists of lexically analysed sentences. Many words have more than one analysis. The task of the dis- ambiguator is to select the contextually appropriate alternative by discarding the improper ones. Some of the inappropriate alternatives can be dis- carded reliably by linguistic rules. For example, we can safely exclude a finite-verb reading if the previ- ous word is an unambiguous determiner. The appli- cation of such rules does not always result in a fully disambiguated output (e.g. adjective-noun ambigui- ties may be left pending) but the amount of ambigu- ity is reduced with next to no errors. Using a large collection of linguistic rules, a lot of ambiguity can be resolved, though some cases remain unresolved. 47 The rule system may also exploit the fact that cer- tain linguistically possible configurations have such a low frequency in certain types of text that they can be ignored. A rule that assumes that a prepo- sition is followed by a noun phrase may be a useful heuristic rule in a practical system, considering that dangling prepositions occur relatively infrequently. Such heuristic rules can be applied to resolve some of the ambiguities that survive the more reliable gram- mar rules. A stochastic disambiguator selects the most likely tag for a word by consulting the neighbouring tags or words, typically in a two or three word window. Because of the limited size of the window, the choices made by a stochastic disambiguator are often quite naive from the linguistic point of view. For instance, the correct resolution of a preposition vs. subordi- nating conjunction ambiguity in a small window is often impossible because both morphological cate- gories can have identical local contexts (for instance, both can be followed by a noun phrase). Some of the errors made by a …

Content truncated to 500 words