Guidelines and Framework for a Large Scale Arabic Diacritized Corpus

Guidelines and Framework for a Large Scale Arabic Diacritized Corpus Wajdi Zaghouani1, Houda Bouamor1, Abdelati Hawwari2, Mona Diab 2, Ossama Obeid1, Mahmoud Ghoneim2, Sawsan Alqahtani2 and Kemal Oflazer1 1Carnegie Mellon University in Qatar {wajdiz,hbouamor,owo}@cmu.edu, ko@cs.cmu.edu 2George Washington University {mdiab,abhawwari,ghoneim,sawsanq}@gwu.edu Abstract This paper presents the annotation guidelines developed as part of an effort to create a large scale manually diacritized corpus for various Arabic text genres. The target size of the annotated corpus is 2 million words. We summarize the guidelines and describe issues encountered during the training of the annotators. We also discuss the challenges posed by the complexity of the Arabic language and how they are addressed. Finally, we present the diacritization annotation procedure and detail the quality of the resulting annotations. Keywords: Arabic Diacritization, Guidelines, Annotation 1. Introduction Written Modern Standard Arabic (MSA) poses many chal- lenges for natural language processing (NLP). Most written Arabic text lacks short vowels and diacritics rendering a mostly consonantal orthography (Schulz, 2004). Arabic di- acritization is an orthographic way to describe Arabic word pronunciation, and to avoid word reading ambiguity. Ara- bic writing system has two classes of symbols: letters and diacritics. diacritics are the marks that reflect the phonolog- ical, morphological and grammatical rules. Diacritization may be classified according to the linguistic rules they are representing, into two types: a) word form diacritization, which shows how a word is pronounced, except the last letter diacritization, and b) case and mood diacritization, which exists above or below the last letter in each word, indicating its grammatical function in the sentence. There are three types of diacritics: vowel, nunation, and shadda (gemination). The lack of diacritics leads usually to considerable lexical and morphological ambiguity as shown in the example in Table 1.1 Full diacritization has been shown to improve state-of-the-art Arabic automatic systems such as speech recognition (ASR) systems (Kirchhoff and Vergyri, 2005) and statistical machine translation (SMT) (Diab et al., 2007). Hence, diacritization has been receiving increased attention in several Arabic NLP applications (Zitouni et al., 2006; Shahrour et al., 2015; Abandah et al., 2015; Belinkov and Glass, 2015). Building models to assign diacritics to each letter in a word requires a large amount of annotated training corpora cover- ing different topics and domains to overcome the sparseness problem. The currently available MSA diacritized corpora are generally limited to newswire stories (those distributed by the LDC), religious texts such as the Holy Quran, or ed- ucational texts. 1We use the Buckwalter transliteration encoding system to rep- resent Arabic in Romanized script (Buckwalter, 2002) Undiacritized Diacritized Buckwalter English ��� ������ /waEad/ he promised ��� ����� /waEodN/ it/a promise ��� ���� �� /wuEida/ he was promised ��� ������� /waEad ˜a/ and he counted ��� ������� /waEud ˜a/ and he was counted Table 1: Possible pronunciations and meanings of the undi- acritized Arabic word ��� /wEd/ This paper presents the work carried out within the opti- mal diacritization scheme for Arabic orthographic repre- sentation (OptDiac) project. The focus of this work is to manually create …

Content truncated to 500 words