A Spelling Correction Corpus for Multiple Arabic Dialects

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4130–4138 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4130 A Spelling Correction Corpus for Multiple Arabic Dialects Fadhl Eryani, Nizar Habash, Houda Bouamor,† Salam Khalifa Computational Approaches to Modeling Language (CAMeL) Lab New York University Abu Dhabi †Carnegie Mellon University in Qatar {fadhl.eryani, nizar.habash, salamkhalifa}@nyu.edu, †hbouamor@qatar.cmu.edu Abstract Arabic dialects are the non-standard varieties of Arabic commonly spoken – and increasingly written on social media – across the Arab world. Arabic dialects do not have standard orthographies, a challenge for natural language processing applications. In this paper, we present the MADAR CODA Corpus, a collection of 10,000 sentences from five Arabic city dialects (Beirut, Cairo, Doha, Rabat, and Tunis) represented in the Conventional Orthography for Dialectal Arabic (CODA) in parallel with their Raw original form. The sentences come from the Multi-Arabic Dialect Applications and Resources (MADAR) Project and are in parallel across the cities (2,000 sentences from each city). This publicly available resource is intended to support research on spelling correction and text normalization for Arabic dialects. We present results on a bootstrapping technique we use to speed up the CODA annotation, as well as on the degree of similarity across the dialects before and after CODA annotation. Keywords: Dialects, Corpora, Spelling Correction, Conventional Orthography for Dialectal Arabic 1. Introduction While the standard form of any language is the variety most likely to receive attention from natural language processing (NLP) researchers and developers, more research is on the rise to address the needs of non-standard varieties and di- alects (Zampieri et al., 2019; Bouamor et al., 2019). The Arabic language, spoken by over 400 million people, is in fact a collective of multiple variants, among which Modern Standard Arabic (MSA) is considered the official primarily written variety of education and culture, even though it is not the native language of any speakers. The other variants are known collectively as Dialectal Arabic (DA), but often classified regionally (as Egyptian, North African, Levan- tine, Gulf, Yemeni) or sub-regionally (i.e, Tunisian, Mo- roccan, Lebanese, and Qatari). Arabic dialects are the true native languages historically connected to Classical Arabic and many other regional languages. These dialects are pri- marily spoken, though their dominance on social media is on the rise. Lacking official recognition, they do not have standard orthographies. As a result, dialectal text tends to have a lot of variety and noise (from a computational lin- guistics point of view). For instance, Habash et al. (2018) reported 27 different spellings for the Egyptian Arabic ut- terance /mabiPulha:S/ “he does not say it”, that vary in terms of etymological or phonetic spelling decisions. This high degree of noise is a major challenge for NLP sys- tem development as it increases the degree of sparsity in the data. Such noise can be handled using modeling techniques that normalize and cluster variants if DA is the input to the system, e.g. in machine translation from dialects to other …

Content truncated to 500 words