A Light Weight Stemmer in Kokborok

A Light Weight Stemmer in Kokborok Braja Gopal Patra, Khumbar Debbarma, Swapan Debbarma Department of Computer Science and Engineering NIT Agartala, India brajagopal.cse@gmail.com, khum_10jan@yahoo.co.in, swapanxavier@gmail.com Dipankar Das, Amitava Das, Sivaji Bandyopadhyay Department of Computer Science and Engineering Jadavpur University, Kolkata, India dipankar.dipnil2005@gmail.com, amitava.santu@gmail.com, sivaji_cse_ju@yahoo.com Abstract Started from the very beginning, Stemming has been playing significant roles in several Natural Language Processing Applications such as information retrieval (IR), machine translation (MT), morph analysis and deciding the part of speech (POS). Several stemmers have been developed for a large number of languages including Indian languages; however no work has been done in Kokborok, a native language of Tripura. In this paper, we have designed a simple rule based stemmer for Kokborok using an affix stripping algorithm. The reduction of inflected words to the stem or root form is performed in the stemmer by stripping the affixes and applying boundary rules where needed. The stemming algorithm has been tested using a corpus of 32578 words and out of which 13044 were uniquely found to have an overall accuracy of 80.02% for minimum suffix stripping algorithm and 85.13% for maximum suffix stripping algorithm. Keywords: Stemming, part of speech (POS), Kokborok, suffix, prefix. [1. Introduction] Kokborok, an Indian language is spoken mainly in the states of Tripura, Assam, Manipur and Mizoram in India and in the neighbouring countries of Myanmar and Bangladesh by more than 2.5 million speakers1 1 http://tripura.nic.in . Kokborok belongs to the Tibeto-Burman (TB) language family. Kokborok shares the genetic features of TB languages that include phonemic tone, widespread stem homophony, subject-object-verb (SOV) word order, agglutinative verb morphology, verb derivational suffixes originating from the semantic bleaching of verbs, duplication or elaboration, evidentiality and emotional attitudes signalled through sentence final particles, aspect rather than tense marking, lack of gender marking and tendency to reduce disyllabic forms to monosyllabic ones. Very specifically, Kokborok has extensive list of suffixes with more limited number of prefixes and different word classes that are formed by affixation of the respective markers. Kokborok is represented either in Roman script or in Bengali script however Bengali script is less preferred as it is difficult to project the actual Proceedings of the Twenty-Fourth Conference on Computational Linguistics and Speech Processing (ROCLING 2012) 318 tonal effect appropriately. The affixes play the most important role in the structure of the language. In Kokborok, the words are formed in three processes called affixation, derivation and compounding .The majority of the roots found in the language are bound and the affixes are the determining factor of the class of the words in the language. Stemming is the process of splitting the stem or root part of the word with its affixes without doing any morphological analysis [6]. Stemming is generally used for Information Retrieval, but is also applied for other Natural Language Processing Applications (NLP) such as Machine Translation (MT), Morph Analysis and Part of Speech (POS) Tagging etc. To the best of our knowledge, at present, there is no such available stemmer in Kokborok …

Content truncated to 500 words