Processing Spontaneous Orthography

Proceedings of NAACL-HLT 2013, pages 585–595, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Processing Spontaneous Orthography Ramy Eskander, Nizar Habash, Owen Rambow, and Nadi Tomeh Center for Computational Learning Systems Columbia University {reskander,habash,rambow,nadi}@ccls.columbia.edu Abstract In cases in which there is no standard or- thography for a language or language vari- ant, written texts will display a variety of or- thographic choices. This is problematic for natural language processing (NLP) because it creates spurious data sparseness. We study the transformation of spontaneously spelled Egyptian Arabic into a conventionalized or- thography which we have previously proposed for NLP purposes. We show that a two-stage process can reduce divergences from this stan- dard by 69%, making subsequent processing of Egyptian Arabic easier. 1 Introduction In areas with diglossia, vernacular spoken variants (“low”) of a language family co-exist with a largely written variant (“high”), which is often not spoken as a native language. Traditionally, the low variants have not been written: written language is reserved for formal occasions and in those formal occasions only the high variant is used. Prototypical exam- ples of diglossia are the German speaking parts of Switzerland, and the Arab world. The advent of the internet has changed linguistic behavior: it is now common to find written informal conversations, in the form of email exchanges, text messages, Twit- ter exchanges, and interactions on blogs and in web forums. These written conversations are typically written in the low variants (or in a mixture of low and high), since conversations in the high variant seem unnatural to the discourse participants. For natural language processing (NLP), this poses many chal- lenges, one of which is the fact that the low vari- ants have not been written much in the past and do not have a standard orthography which is gen- erally agreed on by the linguistic community (and perhaps sanctioned by an authoritative institution). Instead, each discourse participant devises a spon- taneous orthography, in which she chooses among conventions from the high variant to render the spo- ken language. We are thus faced with a large number of ways to spell the same word, none of which can be assumed as “standard” since there is no standard. As a result, the increased data sparseness adds to the challenges of NLP tasks such as machine transla- tion, compared to languages for which orthography is standardized. In this paper, we work on Egyptian Arabic (EGY). We follow the conventions which we have previously proposed for the normalized orthogra- phy for EGY (Habash et al., 2012), called CODA (Conventional Orthography for Dialectal Arabic). In this paper, we investigate how easy it is to con- vert spontaneous orthography of EGY written in Arabic script into CODA orthography automatically. We will refer to this process as “normalization” or “codafication”. We present a freely available system called CODAFY, which we propose as a preproces- sor for NLP modules for EGY. We show that a “do nothing” baseline achieves a normalization perfor- mance of 75.5%, and CODAFY …

Content truncated to 500 words