CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing

CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, Nizar Habash Computational Approaches to Modeling Language (CAMeL) Lab New York University Abu Dhabi, UAE {oobeid,nasser.zalmout,salamkhalifa,dima.taji,mai.oudah, alhafni,go.inoue,fadhl.eryani,ae1541,nizar.habash}@nyu.edu Abstract We present CAMeL Tools, a collection of open-source tools for Arabic natural language processing in Python. CAMeL Tools currently provides utilities for pre-processing, morphological modeling, dialect identification, named entity recognition and sentiment analysis. In this paper, we describe the design of CAMeL Tools and the functionalities it provides. Keywords: Arabic, Open Source, Morphology, Named Entity Recognition, Sentiment Analysis, Dialect Identification 1. Introduction Over the last two decades, there have been many efforts to develop resources to support Arabic natural language pro- cessing (NLP). Some of these resources target specific NLP tasks such as tokenization, diacritization or sentiment anal- ysis, while others attempt to target multiple tasks jointly. Most resources focus on a specific variety of Arabic such as Modern Standard Arabic (MSA), or Egyptian Arabic. These resource also vary in terms of the programming lan- guages they use, the types of interfaces they provide, the data representations and standards they utilize, and the de- gree of public availability (e.g., open-source, commercial, unavailable). These variations make it difficult to write new applications that combine multiple resources. To address many of these issues, we present CAMeL Tools, an open-source Python toolkit that supports Arabic and Arabic dialect pre-processing, morphological modeling, di- alect identification, named entity recognition and sentiment analysis. CAMeL Tools provides command-line interfaces (CLIs) and application programming interfaces (APIs) cov- ering these utilities. The rest of the paper is organized as follows. We present some background on the difficulty of processing Arabic text (Section 2), and then discuss previous work on a variety of Arabic NLP tasks (Section 3). We describe the design and implementation of CAMeL Tools (Section 4) and provide more details about each of its components (Sections 6 to 9). Finally, we discuss some future additions to CAMeL Tools (Section 10). 2. Arabic Linguistics Background Aside of obvious issues such as resource poverty, Arabic poses a number of challenges to NLP: orthographic ambi- guity, morphological richness, dialectal variations, and or- thographic noise (Habash, 2010). While these are not nec- essarily unique issues to Arabic, their combination makes Arabic processing particularly challenging. Orthographic Ambiguity Arabic is generally written us- ing the Arabic Abjad script which uses optional diacritical marks for short vowels and consonantal gemination. While the missing diacritics are not a major challenge to literate native adults, their absence is the main source of ambiguity in Arabic NLP. Morphological Richness Arabic has a rich inflectional morphology system involving gender, number, person, as- pect, mood, case, state and voice, in addition to the cliti- cization of a number of pronouns and particles (conjunc- tions, prepositions, definite article, etc.). This richness leads to MSA verbs with upwards of 5,400 forms. Dialectal Variation While MSA is the official language of culture and education …

Content truncated to 500 words