Exploiting Arabic Diacritization for High Quality Automatic Annotation

Exploiting Arabic Diacritization for High Quality Automatic Annotation Nizar Habash, Anas Shahrour, Muhamed Al-Khalil Computational Approaches to Modeling Language Lab New York University Abu Dhabi, UAE {nizar.habash,anas.shahrour,muhamed.alkhalil}@nyu.edu Abstract We present a novel technique for Arabic morphological annotation. The technique utilizes diacritization to produce morphological annotations of quality comparable to human annotators. Although Arabic text is generally written without diacritics, diacritization is already available for large corpora of Arabic text in several genres. Furthermore, diacritization can be generated at a low cost for new text as it does not require specialized training beyond what educated Arabic typists know. The basic approach is to enrich the input to a state-of-the-art Arabic morphological analyzer with word diacritics (full or partial) to enhance its performance. When applied to fully diacritized text, our approach produces annotations with an accuracy of over 97% on lemma, part-of-speech, and tokenization combined. Keywords: Arabic, Morphology, Annotation, Diacritization 1. Introduction The automatic processing of the Arabic language is chal- lenging for a number of reasons. Two inter-related reasons are Arabic’s complex morphology and its diacritic-optional orthography (Habash, 2010). This combination leads to the high number of about 12.8 possible analyses per word out of context (Shahrour et al., 2015). Much research has been done on Arabic morphological disambiguation and automatic diacritization. Most of this work relies on ex- isting morphologically annotated (i.e., contextually disam- biguated) corpora. Some researchers interested only in di- acritization also use diacritized corpora. The morphologi- cally annotated corpora for Arabic are limited by size and genre. This is because the cost of creating such corpora is prohibitive as time and money must be dedicated to collect data, create guidelines, hire and train annotators, and con- tinuously evaluate the quality of the annotations. In this paper we explore an alternative approach to Ara- bic morphological annotation: we exploit fully diacritized text to rerank the morphological analyses of a state-of-the- art morphological tagger that expects no diacritics as input and provides not only part-of-speech, morphological fea- tures and lemmas, but also diacritizations. We evaluate and analyze the accuracy of this method in generating high quality morphologically analyzed corpora that can them- selves be used for improving tools for Arabic processing, especially for less studied genres. The approach relying solely on diacritization is also very cheap in comparison to hiring specialized (linguist) annotators and training them. Although educated Arabic speakers do not use diacritics in their writing they know how to assign them reasonably well. Of course, we still expect the diacritic annotators to be well educated in Standard Arabic grammar and be fast typists. But the supply and demand, as well as the limited overhead of training is in our favor.1 Obviously, this kind of approach can work only because we already have a mass of 1For reference, a professional Arabic typist can type around 2,250 words/hour undiacritized and 660 words/hour diacritized. S/he can start working immediately to produce diacrtitized text and requires no training or special interfaces. Arabic resources (the taggers and analyzers) that facilitate it. The …

Content truncated to 500 words