A syntax-based part-of-speech analyser

A syntax-based part-of-speech analyser Atro Voutilainen Research Unit for Multilingual Language Technology P.O. Box 4 FIN-00014 University of Helsinki Finland Atro.Voutilainen@Helsinki.FI Abstract There are two main methodologies for constructing the knowledge base of a natural language analyser: the linguis- tic and the data-driven. Recent state-of- the-art part-of-speech taggers are based on the data-driven approach. Because of the known feasibility of the linguis- tic rule-based approach at related levels of description, the success of the data- driven approach in part-of-speech analy- sis may appear surprising. In this paper, a case is made for the syntactic nature of part-of-speech tagging. A new tagger of English that uses only linguistic dis- tributional rules is outlined and empiri- cally evaluated. Tested against a bench- mark corpus of 38,000 words of previ- ously unseen text, this syntax-based sys- tem reaches an accuracy of above 99%. Compared to the 95-97% accuracy of its best competitors, this result suggests the feasibility of the linguistic approach also in part-of-speech analysis. 1 Introduction Part-of-speech analysis usually consists of (i) in- troduction of ambiguity (lexical analysis) and (ii) disambiguation (elimination of illegitimate alter- natives). While introducing ambiguity is regarded as relatively straightforward, disambiguation is known to be a difficult and controversial problem. There are two main methodologies: the linguistic and the data-driven. • In the linguistic approach, the generalisa- tions are based on the linguist's (poten- tially corpus-based) abstractions about the paradigms and syntagms of the language. Distributional generalisations are manually coded as a grammar, a system of constraint rules used for discarding contextually illegit- imate analyses. The linguistic approach is labour-intensive: skill and effort is needed for writing an exhaustive grammar. • In the data-driven approach, frequency-based information is automatically derived from corpora. The learning corpus can consist of plain text, but the best results seem achiev- able with annotated corpora (Merialdo 1994; Elworthy 1994). This corpus-based informa- tion typically concerns sequences of 1-3 tags or words (with some well-known exceptions, e.g. Cutting et al. 1992). Corpus-based infor- mation can be represented e.g. as neural net- works (Eineborg and Gamb/~c k 1994; Schmid 1994), local rules (Brill 1992), or collocational matrices (Garside 1987). In the data-driven approach, no human effort is needed for rule- writing. However, considerable effort may be needed for determining a workable tag set (cf. Cutting 1994) and annotating the training corpus. At the first flush, the linguistic approach may seem an obvious choice. A part-of-speech tagger's task is often illustrated with a noun-verb ambigu- ous word directly preceded by an unambiguous de- terminer (e.g. table in the table). This ambiguity can reliably be resolved with a simple and obvious grammar rule that disallows verbs after determin- ers. Indeed, few contest the fact that reliable linguis- tic rules can be written for resolving some part- of-speech ambiguities. The main problem with this approach seems to be that resolving part-of- speech ambiguities on a large scale, without intro- ducing a considerable error margin, is very diffi- cult at best. At least, no rule-based system with a convincing …

Content truncated to 500 words