Shami: A Corpus of Levantine Arabic Dialects

Shami: A Corpus of Levantine Arabic Dialects Kathrein Abu Kwaik∗, †Motaz Saad, ∗Stergios Chatzikyriakidis, ∗Simon Dobnik ∗CLASP, Department of Philosophy, Linguistics and Theory of Science, Gothenburg University, Sweden †The Islamic University of Gaza, Palestine kathrein.abu.kwaik@gu.se, motaz.saad@gmail.com, stergios.chatzikyriakidis@gu.se, simon.dobnik@gu.se Abstract Modern Standard Arabic (MSA) is the official language used in education and media across the Arab world both in writing and formal speech. However, in daily communication several dialects depending on the country, region as well as other social factors, are used. With the emergence of social media, the dialectal amount of data on the Internet have increased and the NLP tools that support MSA are not well-suited to process this data due to the difference between the dialects and MSA. In this paper, we construct the Shami corpus, the first Levantine Dialect Corpus (SDC) covering data from the four dialects spoken in Palestine, Jordan, Lebanon and Syria. We also describe rules for pre-processing without affecting the meaning so that it is processable by NLP tools. We choose Dialect Identification as the task to evaluate SDC and compare it with two other corpora. In this respect, experiments are conducted using different parameters based on n-gram models and Naive Bayes classifiers. SDC is larger than the existing corpora in terms of size, words and vocabularies. In addition, we use the performance on the Language Identification task to exemplify the similarities and differences in the individual dialects. Keywords: Dialectal Arabic, Levantine Dialect Corpus, Dialect Identification. 1. Introduction Arabic is one of the five most spoken languages in the world; it is spoken by more than 422 million native speakers and used by more than 1.5 billion Muslims.1 The situation in Arabic is a classic case of diglossia, in which the written formal language differs substantially from the spoken ver- nacular. Modern standard Arabic (MSA), which is heavily based on Classical Arabic, is the official written language used in government affairs, news, broadcast media, books and education. MSA is the lingua franca amongst Arabic native speakers. However, the spoken language (collec- tively referred to as Dialectal Arabic) varies widely across the Arab world. The rapid proliferation of social media resulted in these di- alects finding their way to written online social interactions. Dialects of Arabic differ widely among each other and de- pend on the geographic location and the socioeconomic conditions of the speakers. They are commonly categorized in five dominant groups: Maghreb (Libya, Tunisia, Alge- ria, Morocco and western Sahara), Egyptian (Egypt and some parts of Sudan), Levantine (Palestine, Syria, Lebanon and Jordan), Iraqi (Iraq) and Gulf (Qatar, Kuwait, Bahrain, United Arab Emirates, Saudi Arabia, Oman, Yemen) (Zbib et al., 2012). Further varieties can be identified, for exam- ple the Jordanian dialect can be split to Urban, Rural and Bedouin. MSA and Dialectal Arabic share a considerable number of lexical, semantic, syntactic and morphological features, but there are several differences as well. For example, the word ( ����� ¯ayˇs) means “what” in Palestinian but in MSA (������ � �� � ¯ay + ˇsy�) …

Content truncated to 500 words