Creating Resources for Dialectal Arabic from a Single Annotation: A Case Study on Egyptian and Levantine

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3455–3465, Osaka, Japan, December 11-17 2016. Creating Resources for Dialectal Arabic from a Single Annotation: A Case Study on Egyptian and Levantine Ramy Eskander, Nizar Habash†, Owen Rambow and Arfath Pasha Columbia University, USA †New York University Abu Dhabi, UAE rnd2110@columbia.edu, nizar.habash@nyu.edu rambow@ccls.columbia.edu, arfath@ccls.columbia.edu Abstract Arabic dialects present a special problem for natural language processing because there are few Arabic dialect resources, they have no standard orthography, and they have not been studied much. However, as more and more written dialectal Arabic is found on social media, natural lan- guage processing for Arabic dialects has become an important goal. We present a methodology for creating a morphological analyzer and a morphological tagger for dialectal Arabic, and we illustrate it on Egyptian and Levantine Arabic. To our knowledge, these are the first analyzer and tagger for Levantine. 1 Introduction The goal of this paper is to show how a particular type of annotated corpus can be used to create a morphological analyzer and a morphological tagger for a dialect of Arabic, without using any additional resources. A morphological analyzer is a tool that returns all possible morphological analyses for a given input word taken out of any context. A morphological tagger is a tool that identifies the single morphological analysis which is correct for a word given its specific context in a text. We illustrate our work using Egyptian Arabic and Levantine Arabic. Egyptian Arabic is in fact relatively resource-rich compared to other Arabic dialects, but we use a subset of available data to simulate a resource-poor dialect. For Arabic and its dialects, we are faced with a set of well-known challenges (Habash, 2010): they have a rich morphology, the orthography encourages ambiguity, and the dialects do not have standard orthographies (and are generally not well documented linguistically). However, we can also exploit similarities among the dialects and between Dialectal Arabic (DA) and Modern Standard Arabic (MSA), and in fact the writing system, while encouraging ambiguity, also reduces the differences between these variants. The approach we present in this paper is based on the morphological annotation of a text corpus. The annotator provides for each word a conventional orthography, a segmentation, a set of features, and a lemma. We use this information to hypothesize unseen morphological forms, and use all forms to build an analyzer for the new dialect. Because of the close relationship among the variants of Arabic, we also make use of an existing analyzer for MSA, and (in the case of Levantine), an existing Egyptian analyzer. The resulting analyzer is then used in a tagger, which uses the annotated corpus to learn classifiers that choose among all possible analyses. Crucially, we do not require the corpus to be fully annotated, allowing the annotator to concentrate on the most frequent words. We are currently annotating five more dialects with this sort of corpus, with initial corpora available for two dialects (Al-Shargi et al., 2016). The …

Content truncated to 500 words