Automatic Identification of Maghreb Dialects Using a Dictionary-Based Approach

Automatic Identification of Maghreb Dialects Using a Dictionary-Based Approach Houda Saˆadane∗, Hosni Seffih+, Christian Fluhr∗, Khalid Choukri∓, Nasredine Semmar± ∗GEOLSemantics, 12 Avenue Raspail, 94250 Gentilly, France +LIASD, Universit´e Paris 8, 2 rue de la libert´e 93526 Saint-Denis, France ∓ELDA / ELRA , 9 Rue des Cordeli`eres, 75013 Paris, France ±CEA, LIST, Vision and Content Engineering Laboratory, F-91191, Gif-sur-Yvette, France {houda.saadane,hosni.seffih,christian.fluhr}@geolsemantics.com choukri@elda.org, nasredine.semmar@cea.fr Abstract Automatic identification of Arabic dialects in a text is a difficult task, especially for Maghreb languages and when they are written in Arabic or Latin characters (Arabizi). These texts are characterized by the use of code-switching between the Modern Standard Arabic (MSA) and the Arabic Dialect (AD) in the texts written in Arabic, or between Arabizi and foreign languages for those written in Latin. This paper presents the specific resources and tools we have developed for this purpose, with a focus on the transliteration of Arabizi into Arabic (using the dedicated tools for Arabic dialects). A dictionary-based approach to detect the dialectal origin of a text is described, it exhibits satisfactory results. Keywords: Arabic Dialects, Arabizi, Transliteration, Identification of Dialects 1. Introduction The sociolinguistic situation for the Arabic language is characterized by the use of two varieties of one language, which contributes to having a diaglossic conception for this language. This is evidenced by the use of modern stan- dard Arabic (MSA) in the educational, religious and liter- ary fields and in some medias. On the other hand, a large number of dialects are used as mother tongues for many Arabic-speaking populations. In fact, they are the main communication tool spoken in everyday life through infor- mal conversations, exchanges on SMS, forums and social networks, even in e-mails. These dialects vary from one country to another, one region to another, or even from one city to another. In addition, they differ from each other by important phonological, morphological, lexical and syntac- tic characteristics. The processing of informal texts has become an extremely popular field of research among researchers (Yang and Eisenstein, 2013). For Arabic NLP, the identification of dialects is very important and considered as a prepossess- ing step for any natural language application dealing with Arabic language, such as machine translation, information retrieval for social media. It is sometimes considered as a difficult case of language identification where, according to (Zaidan and Callison-Burch, 2011) it is applied to a group of closely related languages that share a common character set. This identification is made even more complex by the absence of orthographic conventions, by the transliteration of Arabic dialects into Latin script (Arabizi) and also the use of code-switching. Recent works have proposed both supervised and unsu- pervised statistical approaches to language identification. However, current methods are based on the assumption that dedicated resources exist, such as large corpora and dictio- naries. Unfortunately, these resources are rarely available for certain languages and dialects, especially for Maghreb dialects. Dialect processing cannot reuse the generic tools and tech- niques that are employed for processing and analyzing MSA texts. …

Content truncated to 500 words