Combining Stochastic and Rule-Based Methods for Disambiguation in Agglutinative Languages

Combining Stochastic and Rule-Based Methods for Disambiguation in Agglutinative Languages Ezeiza N., Alegria I., Arriola J.M., Urizar R. Informatika Fakultatea 649 P.K Donostia E-20080 jibecran@si.ehu.es http://ixa.si.ehu.es Abstract In this paper we present the results of the combination of stochastic and rule-based disambiguation methods applied to Basque languagel. The methods we have used in disambiguation are Constraint Grammar formalism and an HMM based tagger developed within the MULTEXT project. As Basque is an agglutinative language, a morphological analyser is needed to attach all possible readings to each word. Then, CG rules are applied using all the morphological features and this process decreases morphological ambiguity of texts. Finally, we use the MULTEXT project tools to select just one from the possible remaining tags. Using only the stochastic method the error rate is about 14%, but the accuracy may be increased by about 2% enriching the lexi- con with the unknown words. When both methods are combined, the error rate of the whole process is 3.5%. Considering that the training corpus is quite small, that the HMM model is a first order one and that Constraint Grammar of Basque language is still in progress, we think that this com- bined method can achieve good results, and it would be appropriate for other agglutinative languages. Introduction Based on the results of the combination of stochastic and rule-based disambiguation methods applied to Basque language, we will show that the results of the combination are significantly better than the ones obtained applying the methods separately. As Basque is an agglutinative and highly in- This research has been supported by the Education Department of the Government of the Basque Country and the Interministerial Commision for Science and Technology. Aduriz I. UZEI Aldapeta, 20. Donostia E-20009 uzei@sarenet.es fleeted language, a morphological analyser is needed to attach all possible interpretations to each word. This process, which may not be necessary in other languages such as English, makes the tagging task more complex. We use MORFEUS, a robust morphological analyser for Basque developed at the University of the Basque Country (Alegria et al., 1996). We present it briefly in section 1, in the overview of the whole system, the lemmatiser/tagger for Basque EUSLEM. We have added to MOKFEUS a lemma dis- ambiguation process, described in section 2, which discards some of the analyses of the word based on statistical measures. Another important issue concerning a tagger is the tagset itself. We discuss the design of the tagset in section 3. In section 4, we present the results of the ap- plication of rule-based and stochastic disambi- guation methods to Basque. These results are deeply improved by combin- ing both methods as explained in section 5. Finally, we discuss some possible improve- ments of the system and future research. 1 Overview of the system The disambiguation system is integrated in EUSLEM, a lemmatiser/tagger for Basque (Aduriz et al., 1996). EUSLEM has three main modules: • MORFEUS, the morphological analyser based on the two-level formalism. It is a ro- bust and wide …

Content truncated to 500 words