Build Fast and Accurate Lemmatization for Arabic

Build Fast and Accurate Lemmatization for Arabic Hamdy Mubarak QCRI, Hamad Bin Khalifa University (HBKU), Doha, Qatar hmubarak@hbku.edu.qa Abstract In this paper we describe the complexity of building a lemmatizer for Arabic which has a rich and complex morphology, and show some differences between lemmatization and surface stemming, i.e. removing prefixes and suffixes from words. We discuss the need for a fast and accurate lammatization to enhance Arabic Information Retrieval results. We also introduce a new dataset that can be used to test lemmatization accuracy, and an efficient lemmatization algorithm that outperforms state-of-the-art Arabic lemmatization in terms of accuracy and speed. We share the dataset and the code for research purposes. Keywords: Arabic NLP, Lemmatization, Stemming, Information Retrieval, Diactitization 1. Introduction Lemmatization is the process of finding the base form (or lemma) of a word by considering its inflected forms. Lemma is also called dictionary form, or citation form, and it refers to all words having the same meaning. Lemmatization is an important preprocessing step for many applications of text mining and question-answering systems. Researches in Arabic Information Retrieval (IR) systems show the need for representing Arabic words at lemma level for many applications, including keyphrase extraction (El-Shishtawy and Al-Sammak, 2009) and Machine Translation (Dichy and Fargaly, 2003). In addi- tion, lemmatization provides a productive way to generate generic keywords for search engines (SE) or labels for concept maps (Plisson et al., 2004). Word stem is that core part of the word that never changes even with morphological inflections; the part that remains after prefix and suffix removal. Sometimes the stem of the word is different than its lemma, for example the words: believe, believed, believing, and unbelievable share the stem (believ-), and have the normalized word form (believe) standing for the infinitive of the verb (believe). While stemming tries to remove prefixes and suffixes from words that appear with inflections in free text, lemmatiza- tion tries to replace word suffixes with (typically) different suffix to get its lemma. For languages having complex derivational and inflectional morphology, like Arabic, lemmatization needs more than just suffix replacement as will be described in next section. This paper is organized as follows: Section 2. gives some background about Arabic morphology and shows some complexities in building Arabic lemmatization; Section 3. lists IR clustering methods and gives examples to show that lemmatization can enhance search results; Section 4. sur- veys prior work on Arabic stemming and lemmatization; Section 5. introduces the dataset that we created to test lemmatization accuracy; Section 6. describes the algorithm of the system that we built. Results and error analysis are described in section 7.; and Section 8. concludes the paper and lists some tasks for future work. 2. Background Arabic is the largest Semitic language spoken by almost 300 million people. It’s one of the six official languages in the United Nations, and the fifth most widely spoken language after Chinese, Spanish, English, and Hindi1. Arabic has a very rich morphology, both derivational and inflectional. Generally, Arabic words are …

Content truncated to 500 words