Unsupervised Learning of Arabic Stemming Using a Parallel Corpus

Unsupervised Learning of Arabic Stemming using a Parallel Corpus Monica Rogati† Computer Science Department, Carnegie Mellon University mrogati@cs.cmu.edu Scott McCarley IBM TJ Watson Research Center jsmc@watson.ibm.com Yiming Yang Language Technologies Institute, Carnegie Mellon University yiming@cs.cmu.edu Abstract This paper presents an unsupervised learn- ing approach to building a non-English (Arabic) stemmer. The stemming model is based on statistical machine translation and it uses an English stemmer and a small (10K sentences) parallel corpus as its sole training resources. No parallel text is needed after the training phase. Mono- lingual, unannotated text can be used to further improve the stemmer by allow- ing it to adapt to a desired domain or genre. Examples and results will be given for Arabic , but the approach is applica- ble to any language that needs affix re- moval. Our resource-frugal approach re- sults in 87.5% agreement with a state of the art, proprietary Arabic stemmer built using rules, affix lists, and human anno- tated text, in addition to an unsupervised component. Task-based evaluation using Arabic information retrieval indicates an improvement of 22-38% in average pre- cision over unstemmed text, and 96% of the performance of the proprietary stem- mer above. 1 Introduction Stemming is the process of normalizing word vari- ations by removing prefixes and suffixes. From an † Work done while a summer intern at IBM TJ Watson Re- search Center information retrieval point of view, prefixes and suf- fixes add little or no additional meaning; in most cases, both the efficiency and effectiveness of text processing applications such as information retrieval and machine translation are improved. Building a rule-based stemmer for a new, arbitrary language is time consuming and requires experts with linguistic knowledge in that particular lan- guage. Supervised learning also requires large quan- tities of labeled data in the target language, and qual- ity declines when using completely unsupervised methods. We would like to reach a compromise by using a few inexpensive and readily available re- sources in conjunction with unsupervised learning. Our goal is to develop a stemmer generator that is relatively language independent (to the extent that the language accepts stemming) and is trainable us- ing little, inexpensive data. This paper presents an unsupervised learning approach to non-English stemming. The stemming model is based on statisti- cal machine translation and it uses an English stem- mer and a small (10K sentences) parallel corpus as its sole training resources. A parallel corpus is a collection of sentence pairs with the same meaning but in different languages (i.e. United Nations proceedings, bilingual newspa- pers, the Bible). Table 1 shows an example that uses the Buckwalter transliteration (Buckwalter, 1999). Usually, entire documents are translated by humans, and the sentence pairs are subsequently aligned by automatic means. A small parallel corpus can be available when native speakers and translators are not, which makes building a stemmer out of such corpus a preferable direction. Arabic English m$rwE Altqryr Draft report wAkdt mmvlp zAm- byA End ErDhA lltqryr An bldhA y$hd tgyyrAt xTyrp wbEydp Almdy …

Content truncated to 500 words