Large-Coverage Root Lexicon Extraction for Hindi

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 121–129, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Large-Coverage Root Lexicon Extraction for Hindi Cohan Sujay Carlos Monojit Choudhury Microsoft Research IndiaSandipan Dandapat monojitc@microsoft.com Abstract This paper describes a method using mor- phological rules and heuristics, for the au- tomatic extraction of large-coverage lexi- cons of stems and root word-forms from a raw text corpus. We cast the problem of high-coverage lexicon extraction as one of stemming followed by root word-form selection. We examine the use of POS tagging to improve precision and recall of stemming and thereby the coverage of the lexicon. We present accuracy, precision and recall scores for the system on a Hindi corpus. 1 Introduction Large-coverage morphological lexicons are an es- sential component of morphological analysers. Morphological analysers find application in lan- guage processing systems for tasks like tagging, parsing and machine translation. While raw text is an abundant and easily accessible linguistic re- source, high-coverage morphological lexicons are scarce or unavailable in Hindi as in many other languages (Cl´ement et al., 2004). Thus, the devel- opment of better algorithms for the extraction of morphological lexicons from raw text corpora is a task of considerable importance. A root word-form lexicon is an intermediate stage in the creation of a morphological lexicon. In this paper, we consider the problem of extract- ing a large-coverage root word-form lexicon for the Hindi language, a highly inflectional and mod- erately agglutinative Indo-European language spo- ken widely in South Asia. Since a POS tagger, another basic tool, was available along with POS tagged data to train it, and since the error patterns indicated that POS tag- ging could greatly improve the accuracy of the lex- icon, we used the POS tagger in our experiments on lexicon extraction. Previous work in morphological lexicon extrac- tion from a raw corpus often does not achieve very high precision and recall (de Lima, 1998; Oliver and Tadi´c, 2004). In some previous work the pro- cess of lexicon extraction involves incremental or post-construction manual validation of the entire lexicon (Cl´ement et al., 2004; Sagot, 2005; Fors- berg et al., 2006; Sagot et al., 2006; Sagot, 2007). Our method attempts to improve on and extend the previous work by increasing the precision and recall of the system to such a point that manual validation might even be rendered unnecessary. Yet another difference, to our knowledge, is that in our method we cast the problem of lexicon ex- traction as two subproblems: that of stemming and following it, that of root word-form selection. The input resources for our system are as fol- lows: a) raw text corpus, b) morphological rules, c) POS tagger and d) word-segmentation labelled data. We output a stem lexicon and a root word- form lexicon. We take as input a raw text corpus and a set of morphological rules. We first run a stemming algorithm that uses the morphological rules and some heuristics to obtain …

Content truncated to 500 words