Statistical Morphological Analyzer for Hindi

International Joint Conference on Natural Language Processing, pages 1007–1011, Nagoya, Japan, 14-18 October 2013. Statistical Morphological Analyzer for Hindi Deepak Kumar Malladi and Prashanth Mannem Language Technologies Research Center International Institute of Information Technology Hyderabad, AP, India - 500032 {deepak.malladi, prashanth}@research.iiit.ac.in Abstract Morphology is the study of internal struc- ture of words and is an essential early step in many NLP applications such as pars- ing and machine translation. Researchers working in Hindi NLP have either used the widely popular paradigm based analyzer (PBA) or extensions of it. In this work, we undertook a comprehensive evaluation of PBA using the data from the Hindi Tree- bank (HTB) and presented a new morpho- logical analyzer trained on the HTB. Our morphological analyzer has better cover- age and accuracy when compared to the existing analyzers for Hindi. An oracle system that takes the best values from the PBA’s output achieves only 63.41% for lemma, gender, number, person and case. Our statistical analyzer has an accuracy of 84.16% for these morphological attributes when evaluated on the test section of the Hindi Treebank. 1 Introduction Morphological analysis is the task of analyzing the structure of morphemes in a word and is gener- ally a prelude to further complex NLP tasks such as parsing, machine translation, semantic analysis etc. These tasks need an analysis of the words in the sentence in terms of lemma, affixes, parts of speech (POS) etc. Hindi is a morphologically rich language with a relatively free word order. Previous efforts in Hindi morphological analysis concentrated on building rule-based systems that give all the pos- sible analyses for a word form irrespective of its context in the sentence. The paradigm based ana- lyzer (PBA) by Bharati et al. (1995) is one of the most widely used applications among researchers in the Indian NLP community. In paradigm based analysis, words are grouped into a set of paradigms depending on the inflections they take. Each paradigm has a set of add-delete rules to ac- count for its inflections and words belonging to a paradigm take the same inflectional forms. Given a word, the PBA identifies the lemma, coarse POS tag, gender, number, person, case marker, vib- hakti1 and TAM (tense, aspect, modality). Being a rule-based system, the PBA takes a word as in- put and gives all the possible analyses as output. (Table 1 presents an example). It doesn’t pick the correct analysis for a word in its sentential context. Goyal and Lehal’s analyser (2008), which is a re-implementation of the PBA with few ex- tensions, has not done any comparative evalua- tion. Kanuparthi et al. (2012) built a derivational morphological analyzer for Hindi by introduc- ing a layer over the PBA. It identifies 22 deriva- tional suffixes which helps in providing deriva- tional analysis for the word whose suffix matches with one of these 22 suffixes. The large scale machine translation projects2 that are currently under way in India use shallow parser built on PBA and an automatic POS tag- ger. The shallow …

Content truncated to 500 words