Little by Little: Semi Supervised Stemming through Stem Set Minimization

International Joint Conference on Natural Language Processing, pages 774–780, Nagoya, Japan, 14-18 October 2013. Little by Little: Semi Supervised Stemming through Stem Set Minimization Vasudevan N, Pushpak Bhattacharyya Dept. of Computer Science and Engg. IIT Bombay, Mumbai {vasudevan,pb}@cse.iitb.ac.in Abstract In this paper we take an important step to- wards completely unsupervised stemming by giving a scheme for semi supervised stemming. The input to the system is a list of word forms and suffixes. The motivation of the work comes from the need to create a root or stem identifier for a language that has electronic corpora and some elementary linguistic work in the form of, say, suffix list. The scope of our work is suffix based morphology, (i.e., no prefix or infix morphology). We give two greedy algorithms for stemming. We have performed extensive experimen- tation with four languages: English, Hindi, Malayalam and Marathi. Accuracy figures ranges from 80% to 88% are reported for all languages. 1 Introduction Stemming is critical for many NLP, IR and IE problems (Hull, 1996). In the current paper, we report construction of a semi supervised stemmer that does stemming by minimizing the total num- ber of distinct stems. The input to the system is the word list along with the legal suffix list of the lan- guage. Even if a language does not have an elab- orate linguistic tradition and exhaustive body of linguistic work, the language is expected to have at least the legal suffix list for nouns and verbs. To get the intuition behind our work, consider the word list {boy, boys, moss, mosses}. The split- ting (the split is formally defined later) that gener- ate the minimum number of stems (viz., 2) from the above word list is {boy+φ, boy+s, moss+φ, moss+es }, where φ is the null suffix. The mini- mum stem set is {boy, moss}. Any other splitting, say mosse+s will increase the number of stems. This work is applicable to languages with con- catenative morphology where suffixes stack one after another. However, problems arise when there are phonemic changes in the boundaries of the stems and suffixes (sandhi). In such situation, ex- istence of fused, composite suffixes in the suffix list is assumed. The roadmap of the paper is as follows. Related work in morphology learning is explained in sec- tion 2. Notations and terminologies used in this paper are defined in section 3. In section 4 we defined the stemming problem addressed in this work. Two models for this stemming are proposed in section 5 and section 6. In section 7, we de- scribed various experiments conducted. The con- clusions and future works are presented in section 8. 2 Related Work Morphology learning is one of the widely at- tempted problems in NLP. A recent survey by Har- ald Hammarstr¨om (2011) gives an overall view of unsupervised morphology learning. The Linguis- tica (Goldsmith, 2001) model based on minimum description length (MDL) principle is one of the benchmark works of unsupervised stemming. In the Linguistica model, the authors …

Content truncated to 500 words