Automatic Transliteration of Romanized Dialectal Arabic

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 30–38, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics Automatic Transliteration of Romanized Dialectal Arabic Mohamed Al-Badrashiny†, Ramy Eskander, Nizar Habash and Owen Rambow †Department of Computer Science, The George Washington University, Washington, DC †badrashiny@gwu.edu Center for Computational Learning Systems, Columbia University, NYC, NY {reskander,habash,rambow}@ccls.columbia.edu Abstract In this paper, we address the problem of converting Dialectal Arabic (DA) text that is written in the Latin script (called Arabizi) into Arabic script following the CODA convention for DA orthography. The presented system uses a finite state transducer trained at the character level to generate all possible transliterations for the input Arabizi words. We then filter the generated list using a DA morpholog- ical analyzer. After that we pick the best choice for each input word using a lan- guage model. We achieve an accuracy of 69.4% on an unseen test set compared to 63.1% using a system which represents a previously proposed approach. 1 Introduction The Arabic language is a collection of varieties: Modern Standard Arabic (MSA), which is used in formal settings and has a standard orthogra- phy, and different forms of Dialectal Arabic (DA), which are commonly used informally and with in- creasing presence on the web, but which do not have standard orthographies. While both MSA and DA are commonly written in the Arabic script, DA (and less so MSA) is sometimes written in the Latin script. This happens when using an Ara- bic keyboard is dispreferred or impossible, for ex- ample when communicating from a mobile phone that has no Arabic script support. Arabic written in the Latin script is often referred to as “Arabizi”. Arabizi is not a letter-based transliteration from the Arabic script as is, for example, the Buck- walter transliteration (Buckwalter, 2004). Instead, roughly speaking, writers use sound-to-letter rules inspired by those of English1 as well as informally 1In different parts of the Arab World, the basis for the Latin script rendering of DA may come from different lan- established conventions to render the sounds of the DA sentence. Because the sound-to-letter rules of English are very different from those of Ara- bic, we obtain complex mappings between the two writing systems. This issue is compounded by the underlying problem that DA itself does not have any standard orthography in the Arabic script. Ta- ble 1 shows different plausible ways of writing an Egyptian Arabic (EGY) sentence in Arabizi and in Arabic script. Arabizi poses a problem for natural language processing (NLP). While some tools have recently become available for processing EGY input, e.g., (Habash et al., 2012b; Habash et al., 2013; Pasha et al., 2014), they expect Arabic script input (or a Buckwalter transliteration). They cannot process Arabizi. We therefore need a tool that converts from Arabizi to Arabic script. However, the lack of standard orthography in EGY compounds the problem: what should we convert Arabizi into? Our answer to this question is to use CODA, a conventional orthography created …

Content truncated to 500 words