Learning from Relatives: Unified Dialectal Arabic Segmentation

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 432–441, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Learning from Relatives: Unified Dialectal Arabic Segmentation Younes Samih1, Mohamed Eldesouki3, Mohammed Attia2, Kareem Darwish3, Ahmed Abdelali3, Hamdy Mubarak3, and Laura Kallmeyer1 1Dept. of Computational Linguistics,University of D¨usseldorf, D¨usseldorf, Germany 2Google Inc., New York City, USA 3Qatar Computing Research Institute, HBKU, Doha, Qatar 1{samih,kallmeyer}@phil.hhu.de 2attia@google.com 3{mohamohamed,hmubarak,aabdelali,kdarwish}@hbku.edu.qa Abstract Arabic dialects do not just share a com- mon koin´e, but there are shared pan- dialectal linguistic phenomena that allow computational models for dialects to learn from each other. In this paper we build a unified segmentation model where the training data for different dialects are combined and a single model is trained. The model yields higher accuracies than dialect-specific models, eliminating the need for dialect identification before seg- mentation. We also measure the degree of relatedness between four major Ara- bic dialects by testing how a segmenta- tion model trained on one dialect performs on the other dialects. We found that lin- guistic relatedness is contingent with ge- ographical proximity. In our experiments we use SVM-based ranking and bi-LSTM- CRF sequence labeling. 1 Introduction Segmenting Arabic words into their constituent parts is important for a variety of applications such as machine translation, parsing and information retrieval. Though much work has focused on seg- menting Modern Standard Arabic (MSA), recent work began to examine dialectal segmentation in some Arabic dialects. Dialectal segmentation is becoming increasingly important due to the ubiq- uity of social media, where users typically write in their own dialects as opposed to MSA. Dialec- tal text poses interesting challenges such as lack of spelling standards, pervasiveness of word merg- ing, letter substitution or deletion, and foreign word borrowing. Existing work on dialectal seg- mentation focused on building resources and tools for each dialect separately (Habash et al., 2013; Pasha et al., 2014; Samih et al., 2017). The ratio- nal for the separation is that different dialects have different affixes, make different lexical choices, and are influenced by different foreign languages. However, performing reliable dialect identifica- tion to properly route text to the appropriate seg- menter may be problematic, because conventional dialectal identification may lead to results that are lower than 90% (Darwish et al., 2014). Thus, building a segmenter that performs reliably across multiple dialects without the need for dialect iden- tification is desirable. In this paper we examine the effectiveness of us- ing a segmenter built for one dialect in segmenting other dialects. Next, we explore combining train- ing data for different dialects in building a joint segmentation model for all dialects. We show that the joint segmentation model matches or outper- forms dialect-specific segmentation models. For this work, we use training data in four different di- alects, namely Egyptian (EGY), Levantine (LEV), Gulf (GLF), and Maghrebi (MGR). We utilize two methods for segmentation. The first poses seg- mentation as a ranking problem, where we use an SVM ranker. The second poses …

Content truncated to 500 words