Hierarchical Aggregation of Dialectal Data for Arabic Dialect Identification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4586–4596 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4586 Hierarchical Aggregation of Dialectal Data for Arabic Dialect Identification Nurpeiis Baimukan, Houda Bouamor†, Nizar Habash Computational Approaches to Modeling Language (CAMeL) Lab New York University Abu Dhabi †Carnegie Mellon University in Qatar {nurpeiis,nizar.habash}@nyu.edu hbouamor@cmu.edu Abstract Arabic is a collection of dialectal variants that are historically related but significantly different. These differences can be seen across regions, countries, and even cities in the same countries. Previous work on Arabic Dialect identification has focused mainly on specific dialect levels (region, country, province, or city) using level-specific resources; and different efforts used different schemas and labels. In this paper, we present the first effort aiming at defining a standard unified three-level hierarchical schema (region-country-city) for dialectal Arabic classification. We map 29 different data sets to this unified schema, and use the common mapping to facilitate aggregating these data sets. We test the value of such aggregation by building language models and using them in dialect identification. We make our label mapping code and aggregated language models publicly available. Keywords: Arabic Dialects, Dialect Identification, Language Models 1. Introduction Dialect identification (DID) is a natural language pro- cessing (NLP) task that aims at automatically determin- ing the dialect of a given speech fragment or text (Et- man and Beex, 2015). Since dialectal difference tend to be more subtle in relative terms to language differ- ences, the DID task is harder than language identifica- tion. In this paper we focus on Arabic dialect identifi- cation, but we believe that our techniques and insights are extensible to other languages and dialect groups. Arabic is a collection of dialectal variants that are his- torically related but significantly different. Arabic di- alects are often classified in terms of geography at different levels of granularity: at the regional, coun- try, province and city levels (Zaidan and Callison- Burch, 2012; Bouamor et al., 2019; Abdul-Mageed et al., 2020; Abdul-Mageed et al., 2021). Dialects are different from each other and Modern Standard Ara- bic (MSA) in terms of phonology, orthography, mor- phology, and the lexicon. Arabic speakers tend to code-switch between their dialect and MSA, creating sentences with different levels/percentages of “dialect- ness” (Habash et al., 2008; Elfardy et al., 2013; El-Haj et al., 2018; Ali et al., 2021). That said, many of the differences are not observed in the written forms since in Arabic orthography, writing vowel diacritics is op- tional. Several efforts have targeted creating different re- sources at different hierarchical levels going from cities to regions (Zaidan and Callison-Burch, 2011; Sma¨ıli et al., 2014; Jarrar et al., 2016; Al-Twairesh et al., 2018; Bouamor et al., 2018; Abdul-Mageed et al., 2020). Most of these resources were built in independent ef- forts using different labeling schemas, making the joint use and comparison of these data sets impractical. In this paper, we address this issue by defining a unified hierarchical schema for dialectal Arabic …

Content truncated to 500 words