You Tweet What You Speak: A City-Level Dataset of Arabic Dialects

You Tweet What You Speak: A City-Level Dataset of Arabic Dialects Muhammad Abdul-Mageed, Hassan Alhuzali, Mohamed Elaraby Natural Language Processing Lab University of British Columbia muhammad.mageeed@ubc.ca, {halhuzali,mohamed.elaraby}@alumni.ubc.ca Abstract Arabic has a wide range of varieties or dialects. Although a number of pioneering works have targeted some Arabic dialects, other dialects remain largely without investigation. A serious bottleneck for studying these dialects is lack of any data that can be exploited in computational models. In this work, we aim to bridge this gap: We present a considerably large dataset of > 1/4 billion tweets representing a wide range of dialects. Our dataset is more nuanced than previously reported work in that it is labeled at the fine-grained level of city. More specifically, the data represent 29 major Arab cities from 10 Arab countries with varying dialects (e.g., Egyptian, Gulf, KSA, Levantine, Yemeni). 1. Introduction The Arab world covers a vast region across the two con- tents, Africa and Asia. The term Arabic itself refers to a collection of varieties, possibly comprised by three major categories: (1) Modern Standard Arabic (MSA), (2) Clas- sical Arabic (CA), and (3) Dialectal Arabic (DA). MSA (Badawi, 1973) is the modern variety of the language used in educational settings and some pan-Arab networks like AlJazeera (Abdul-Mageed, 2008; Abdul-Mageed and Her- ring, 2008). CA is the language of the Qura’an (the Holy Book of Islam) that is employed in religious and elite liter- ary works. MSA and CA differ mainly lexically and mor- phologically, with fewer structural and syntactic differences (Bateson, 1967; Ryding, 2005). DA is a collection of arbi- trarily defined (Versteegh, 2001; Habash, 2010) variations, although geography does play a role in the classification of DA. Most Arabic varieties remained primarily spoken for a long time. With the advent of the internet and the prolifera- tion of social media, Arabic dialects found their way to online written form (Abdul-Mageed, 2015). Early compu- tational studies of Arabic dialects have depended on data collected from blogs and comments on online news sites, e.g., (Diab et al., 2010; Elfardy and Diab, 2012). Due to the costly efforts associated with labeling the data with dialect tags, these pioneering works have focused on a few vari- eties like Egyptian or Levantine (Elfardy and Diab, 2013; Zaidan and Callison-Burch, 2011; Zaidan and Callison- Burch, 2014). User-provided location information in the Twitter microblogging platform have made it possible to collect data with nuanced geographical labels in ways not previously possible, see e.g., (Jurgens et al., 2017). We de- pend on these cues to label our dataset with location tags as a proxy for the relevant dialects. Although no agreement exists as to where dialectal bound- aries should be drawn, some proposals have been made. Figure 1 shows only one such classifications of Arabic di- alects. The vast geographic extension the Arab world con- stitutes naturally translates into rich and varied linguistic tradition, thus making nuanced study of Arabic dialects an attractive object of scientific investigation. In this work, we take a step …

Content truncated to 500 words