High-Performance, Language-Independent Morphological Segmentation

Proceedings of NAACL HLT 2007, pages 155–163, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics High-Performance, Language-Independent Morphological Segmentation Sajib Dasgupta and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {sajib,vince}@hlt.utdallas.edu Abstract This paper introduces an unsupervised morphological segmentation algorithm that shows robust performance for four languages with different levels of mor- phological complexity. In particular, our algorithm outperforms Goldsmith’s Lin- guistica and Creutz and Lagus’s Mor- phessor for English and Bengali, and achieves performance that is comparable to the best results for all three PASCAL evaluation datasets. Improvements arise from (1) the use of relative corpus fre- quency and suffix level similarity for de- tecting incorrect morpheme attachments and (2) the induction of orthographic rules and allomorphs for segmenting words where roots exhibit spelling changes dur- ing morpheme attachments. 1 Introduction Morphological analysis is the task of segmenting a word into morphemes, the smallest meaning- bearing elements of natural languages. Though very successful, knowledge-based morphological analyzers operate by relying on manually designed segmentation heuristics (e.g. Koskenniemi (1983)), which require a lot of linguistic expertise and are time-consuming to construct. As a result, research in morphological analysis has exhibited a shift to unsupervised approaches, in which a word is typi- cally segmented based on morphemes that are automatically induced from an unannotated corpus. Unsupervised approaches have achieved consider- able success for English and many European lan- guages (e.g. Goldsmith (2001), Schone and Juraf- sky (2001), Freitag (2005)). The recent PASCAL Challenge on Unsupervised Segmentation of Words into Morphemes1 has further intensified interest in this problem, selecting as target lan- guages English as well as two highly agglutinative languages, Turkish and Finnish. However, the evaluation of the Challenge reveals that (1) the success of existing unsupervised morphological parsers does not carry over to the two agglutinative languages, and (2) no segmentation algorithm achieves good performance for all three languages. Motivated by these state-of-the-art results, our goal in this paper is to develop an unsupervised morphological segmentation algorithm that can work well across different languages. With this goal in mind, we evaluate our algorithm on four languages with different levels of morphological complexity, namely English, Turkish, Finnish and Bengali. It is worth noting that Bengali is an under- investigated Indo-Aryan language that is highly inflectional and lies between English and Turk- ish/Finnish in terms of morphological complexity. Experimental results demonstrate the robustness of our algorithm across languages: it not only outper- forms Goldsmith’s (2001) Linguistica and Creutz and Lagus’s (2005) Morphessor for English and Bengali, but also compares favorably to the best- performing PASCAL morphological parsers when evaluated on all three datasets in the Challenge. The performance improvements of our segmen- tation algorithm over existing morphological ana- lyzers can be attributed to our extending Keshava and Pitler’s (2006) segmentation method, the best performer for English in the aforementioned 1 http://www.cis.hut.fi/morphochallenge2005/ 155 PASCAL Challenge, with the capability of han- dling two under-investigated problems: Detecting incorrect attachments. Many existing morphological parsers incorrectly segment “candi- date” as …

Content truncated to 500 words