PurePos 2.0: a hybrid tool for morphological disambiguation

Proceedings of Recent Advances in Natural Language Processing, pages 539–545, Hissar, Bulgaria, 7-13 September 2013. PurePos 2.0: a hybrid tool for morphological disambiguation Gy¨orgy Orosz and Attila Nov´ak P´azm´any P´eter Catholic University, Faculty of Information Technology MTA-PPKE Natural Language Processing Group 50/a Pr´ater street, Budapest, Hungary {oroszgy, novak.attila}@itk.ppke.hu Abstract We present PurePos, an open-source HMM-based automatic morphological an- notation tool. PurePos can perform tag- ging and lemmatization at the same time, it is very fast to train, with the possibil- ity of easy integration of symbolic rule- based components into the annotation pro- cess that can be used to boost the accu- racy of the tool. The hybrid approach im- plemented in PurePos is especially ben- eficial in the case of rich morphology, highly detailed annotation schemes and if a small amount of training data is avail- able. Evaluation of the tool was on a Hun- garian corpus revealed that its hybrid com- ponents significantly improve overall an- notation accuracy. 1 Introduction Part-of-speech tagging is one of the basic and commonly studied tasks of natural language pro- cessing. High accuracy of morphosyntactic anno- tation is crucial since tagging is usually part of a language processing pipeline, thus tagging errors propagate. Several PoS tagging tools have been created and made available during the years, how- ever, PoS tagging is just the subtask of morpholog- ical annotation: in addition to the morphosyntactic tag, the lemma needs to be identified for each to- ken. For morphologically not very rich languages, like English, a cascade of a tagger and a stem- mer may yield an acceptable performance, but in the case of morphologically rich languages, incor- porating morphological knowledge in the form of a morphological analyzer (MA) into the tagging process seems to be necessary not only to obtain high tagging accuracy but also to provide correct lemmata. Sequence tagging tasks are often solved us- ing statistical modelling techniques, since hav- ing a huge amount of annotated data, a decent method can learn important regularities, and ap- plying this knowledge can yield highly accurate results. Smoothing techniques are commonly used in statistical natural language processing applica- tions to alleviate problems caused by data sparse- ness. However, this prevents purely statistical models from being able to exclude events from the model that are known to be impossible to oc- cur. Rule-based tools can find their niche here: one can either use rules to filter out agrammatical sequences, or ones that do not occur in a given do- main. Hybrid methods combining statistical and rule-based approaches are getting more and more popular, since these are often able to yield a level of performance not attainable by either the statis- tical or the rule-based component alone. In this paper, we describe the improvements that we made to an open source tool, PurePos, which, combining statistical models with symbolic and rule-based components, can generate highly accu- rate morphological annotation. Our paper is struc- tured as follows. First, we motivate the model with annotation scenarios where a hybrid approach …

Content truncated to 500 words