Arabic Dialect Identification in the Context of Bivalency and Code-Switching

Arabic Dialect Identification in the Context of Bivalency and Code-Switching Mahmoud El-Haj1, Paul Rayson1 and Mariam Aboelezz2 1School of Computing and Communications, Lancaster University, UK and 2British Library, UK {m.el-haj, p.rayson}@lancaster.ac.uk, mariam.aboelezz@bl.uk Abstract In this paper we use a novel approach towards Arabic dialect identification using language bivalency and written code-switching. Bivalency between languages or dialects is where a word or element is treated by language users as having a fundamentally similar semantic content in more than one language or dialect. Arabic dialect identification in writing is a difficult task even for humans due to the fact that words are used interchangeably between dialects. The task of automatically identifying dialect is harder and classifiers trained using only n-grams will perform poorly when tested on unseen data. Such approaches require significant amounts of annotated training data which is costly and time consuming to produce. Currently available Arabic dialect datasets do not exceed a few hundred thousand sentences, thus we need to extract features other than word and character n-grams. In our work we present experimental results from automatically identifying dialects from the four main Arabic dialect regions (Egypt, North Africa, Gulf and Levant) in addition to Standard Arabic. We extend previous work by incorporating additional grammatical and stylistic features and define a subtractive bivalency profiling approach to address issues of bivalent words across the examined Arabic dialects. The results show that our new methods classification accuracy can reach more than 76% and score well (66%) when tested on completely unseen data. Keywords: Arabic, bivalency, language identification, dialects, machine learning, NLP 1. Introduction In natural language processing, the problem of detecting the language of a given text is called language identifica- tion or language guessing. In early work, relatively simple approaches employing character n-grams proved to be suc- cessful (Cavnar and Trenkle, 1994; Dunning, 1994; Souter et al., 1994). More recently, this has been seen as a clas- sification problem where machine learning is used to dis- tinguish between languages (Gupta et al., 2015). The vast majority of language identification research has focused on differentiating between languages. In this paper, we instead focus on differentiating regional varieties of the same lan- guage (i.e. dialects), taking Arabic as our case study. Automatically identifying dialects could prove fruitful in fields such as natural language processing, corpus linguis- tics and machine translation (Zaidan and Callison-Burch, 2014). The process of hiring human participants to iden- tify dialects is very costly and it is a time consuming job. Therefore, machine automation could work as a quick and cheap alternative provided we can create an effective mix of new methods with the appropriate dataset. In this study, we tackle the problem of automatically identi- fying Arabic dialects using a variety of approaches in order to address bivalency and dialectal written code-switching (Habash et al., 2008; Biadsy et al., 2009) which pose sig- nificant challenges for existing approaches. We apply our novel Subtractive Bivalency Profiling (SBP) approach to address the issue of bivalent words across the Arabic di- alects …

Content truncated to 500 words