AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic

Proceedings of the 19th Conference on Computational Language Learning, pages 42–51, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic Mohamed Al-Badrashiny, Heba Elfardy† and Mona Diab Department of Computer Science, The George Washington University {badrashiny,mtdiab}@gwu.edu †Department of Computer Science, Columbia University †heba@cs.columbia.edu Abstract In this paper, we present a hybrid approach for performing token and sentence levels Dialect Identification in Arabic. Specifi- cally we try to identify whether each to- ken in a given sentence belongs to Modern Standard Arabic (MSA), Egyptian Dialec- tal Arabic (EDA) or some other class and whether the whole sentence is mostly EDA or MSA. The token level component re- lies on a Conditional Random Field (CRF) classifier that uses decisions from several underlying components such as language models, a named entity recognizer and and a morphological analyzer to label each word in the sentence. The sentence level component uses a classifier ensemble sys- tem that relies on two independent under- lying classifiers that model different as- pects of the language. Using a feature- selection heuristic, we select the best set of features for each of these two classifiers. We then train another classifier that uses the class labels and the confidence scores generated by each of the two underlying classifiers to decide upon the final class for each sentence. The token level compo- nent yields a new state of the art F-score of 90.6% (compared to previous state of the art of 86.8%) and the sentence level com- ponent yields an accuracy of 90.8% (com- pared to 86.6% obtained by the best state of the art system). 1 Introduction In this age of social media ubiquity, we note the pervasive presence of informal language mixed in with formal language. Degree of mixing formal and informal language registers varies across lan- guages making it ever harder to process. The prob- lem is quite pronounced in Arabic where the dif- ference between the formal modern standard Ara- bic (MSA) and the informal dialects of Arabic (DA) could add up to a difference in language morphologically, lexically, syntactically, seman- tically and pragmatically, exacerbating the chal- lenges for almost all NLP tasks. MSA is used in formal settings, edited media, and education. On the other hand the spoken, and, currently writ- ten in social media and penetrating formal me- dia, are the informal vernaculars. There are mul- tiple dialects corresponding to different parts of the Arab world: (1) Egyptian, (2) Levantine, (3) Gulf, (4) Moroccan, and, (5) Iraqi. For each one of these sub-dialectal variants exist. Speakers/writers code switch between the two forms of the lan- guage especially in social media text both inter and intra sententially. Automatically identifying code-switching between variants of the same lan- guage (Dialect Identification) is quite challeng- ing due to the lexical overlap and significant se- mantic and pragmatic variation yet it is crucial as a preprocessing step before building any Ara- bic NLP tool. MSA trained tools …

Content truncated to 500 words