CODACT: Towards Identifying Orthographic Variants in Dialectal Arabic

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 318–326, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP CODACT: Towards Identifying Orthographic Variants in Dialectal Arabic Pradeep Dasigi Computer Science Department Columbia University in the City of New York pd2359@columbia.edu Mona Diab Center for Computational Learning Systems Columbia University in the City of New York mdiab@ccls.columbia.edu Abstract Dialectal Arabic (DA) is the spoken ver- nacular for over 300M people worldwide. DA is emerging as the form of Arabic written in online communication: chats, emails, blogs, etc. However, most existing NLP tools for Arabic are designed for pro- cessing Modern Standard Arabic, a vari- ety that is more formal and scripted. Apart from the genre variation that is a hindrance for any language processing, even in En- glish, DA has no orthographic standard, compared to MSA that has a standard or- thography and script. Accordingly, a word may be written in many possible incon- sistent spellings rendering the processing of DA very challenging. To solve this problem, such inconsistencies have to be normalized. This work is the first step towards addressing this problem, as we attempt to identify spelling variants in a given textual document. We present an unsupervised clustering approach that ad- dresses the problem of identifying ortho- graphic variants in DA. We employ differ- ent similarity measures that exploit string similarity and contextual semantic similar- ity. To our knowledge this is the first at- tempt at solving the problem for DA. Our approaches are tested on data in two di- alects of Arabic - Egyptian and Levan- tine. Our system achieves the highest En- tropy of 0.19 for Egyptian (corresponding to 68% cluster precision) and Levantine (corresponding to 64% cluster precision) respectively. This constitutes a significant reduction in entropy (from 0.47 for Egyp- tian and 0.51 for Levantine) and improve- ment in cluster precision (from 29% for both) from the baseline. 1 Introduction Arabic is the native tongue of over 300M peo- ple world wide. The Arabic language exhibits a relatively unique linguistic phenomena known as diglossia (Ferguson, 1959) where two forms of the language live side by side: a standard formal form known as Modern Standard Arabic (MSA) and an informal spoken form, the vernaculars used in everyday communication referred to as Dialec- tal Arabic (DA). MSA is the only language of ed- ucation and is used in formal settings and Broad- cast news. The only written standard is in MSA using the Arabic script. Technically there are no native speakers of MSA. On the other hand, DA is the mother tongue for all native speakers of Ara- bic however it is not traditionally a written form of the language and it differs significantly enough from MSA on all levels of linguistic representation that results in huge inconsistencies in orthography. This was not a problem a decade ago from an NLP perspective since all the resources were in MSA. Now with the proliferation of online media and informal genres, DA is ubiquitous online. …

Content truncated to 500 words