Improving Brill’s POS Tagger for an Agglutinative Language

IMPROVING BRILL'S POS TAGGER FOR AN AGGLUTINATIVE LANGUAGE Be~ta Megyesi Stockholm University Department of Linguistics Computational Linguistics S- 10691 Stockholm, Sweden bea@ling.su.se Abstract In this paper Brill's rule-based PoS tagger is tested and adapted for Hungarian. It is shown that the present system does not obtain as high accuracy for Hungarian as it does for English (and other Germanic languages) because of the structural difference between these languages. Hungarian, unlike English, has rich morphology, is agglutinative with some inflectional characteristics and has fairly free word order. The tagger has the greatest difficulties with parts-of-speech belonging to open classes because of their complicated morphological structure. It is shown that the accuracy of tagging can be increased from approximately 83% to 97% by simply changing the rule generating mechanisms, namely the lexical templates in the lexical training module. 1. Introduction In 1992 Eric Brill presented a rule-based tagging system which differs from other rule-based systems because it automatically infers rules from a training corpus. The tagger does not use hand-crafted rules or prespecified language information, nor does the tagger use external lexicons. According to Brill (1992) 'there is a very small amount of general linguistic knowledge built into the system, but no language-specific knowledge'. The grammar is induced directly from the training corpus without human intervention or expert knowledge. The only additional component necessary is a small, manually and correctly annotated corpus - the training corpus - which serves as input to the tagger. The system is then able to derive lexical/morphological and contextual information from the training corpus and 'learns' how to deduce the most likely part of speech tag for a word. Once the training is completed, the tagger can be used to annotate new, unannotated corpora based on the tag set of the training corpus. The tagger has been trained for tagging English texts with an accuracy of 97% (Brill, 1994). In this study Brill's rule-based part of speech (PoS) tagger is tested on Hungarian, a dissimilar language, concerning both morphology and syntax, to English. The main goal is i) to find out if Brill's system is immediately applicable to a language, which greatly differs in structure from English, with a high degree of accuracy and (if not) ii) to improve the training strategies to better fit for agglutinative/inflectional languages with a complex morphological structure. Hungarian is basically agglutinative, i.e. grammatical relations are expressed by means of affixes. Hungarian is also inflectional; it is difficult to assign 275 morphemes precisely to the different parts of the affixes. The morphotactics of the possible forms is very regular. For example, Hungarian nouns may be analyzed as a stem followed by three positions in which inflectional suffixes (for number, possessor and case) can occur. Additionally, derivational suffixes, which change the PoS of a word, are very common and productive. Verbs, nouns, adjectives and even adverbs can be further derived. Thus, a stem can get one or more derivational and often several inflectional suffixes. For example, the word taldlataiknak 'of their hits' consists …

Content truncated to 500 words