Capturing Out-of-Vocabulary Words in Arabic Text

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 258–266, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Capturing Out-of-Vocabulary Words in Arabic Text Abdusalam F.A. Nwesri S.M.M. Tahaghoghi Falk Scholer School of Computer Science and Information Technology RMIT University, GPO Box 2476V, Melbourne 3001, Australia {nwesri,saied,fscholer}@cs.rmit.edu.au Abstract The increasing flow of information be- tween languages has led to a rise in the fre- quency of non-native or loan words, where terms of one language appear transliter- ated in another. Dealing with such out of vocabulary words is essential for suc- cessful cross-lingual information retrieval. For example, techniques such as stemming should not be applied indiscriminately to all words in a collection, and so before any stemming, foreign words need to be iden- tified. In this paper, we investigate three approaches for the identification of foreign words in Arabic text: lexicons, language patterns, and n-grams and present that re- sults show that lexicon-based approaches outperform the other techniques. 1 Introduction Arabic words are derived from roots having three, four, or, in rare instances, five characters. The derivation process consistently follows patterns that are based on the three letter verb � � � � � � � (/fa�ala/ to do)1. Each root word matches a base pattern. Characters are added at the beginning, the mid- dle, or end of the root, but the base characters that match the pattern remain unchanged. The pronunciation of Arabic characters is as- sociated with short vowels; these are represented by diacritics, and attached to other characters to show how the characters should be pronounced. An Arabic character can be pronounced in several different ways. For example, the letter H . with the 1We represent phonetics using the International Pho- netic Alphabet (http://www.arts.gla.ac.uk/IPA/ ipachart.html) diacritic Fatha � H . is pronounced / ba/, with the dia- critic Kasra H . � is pronounced / bI/, and with having the diacritic Damma � H . is pronounced / bU/. Di- acritics are not shown in general written Arabic, and the reader must rely on the context to deter- mine the implicit diacritics and therefore the pro- nunciation of each word. For example, the word I . � � X can represent � I . � � � � X (/Dahaba/ = went), � I . � � � � X (/Dahab/ = gold). Pure Arabic words follow restricted rules in their construction to keep them short and easy to pronounce. Their sounds usually follow the CVCV pattern, where C stands for a consonant and V stands for a Vowel. An Arabic word never has two consecutive consonants nor four consecu- tive vowels (Al-Shanti, 1996). Foreign words are words that are borrowed from other languages. Some are remodelled to con- form with Arabic word paradigms, and become part of the Arabic lexicon; others are transliterated into Arabic as they are pronounced by different Arabic speakers, with some segmental and vowel changes. The latter are called Out-Of-Vocabulary (OOV) words as they are not …

Content truncated to 500 words