Diacritization for Real-World Arabic Texts

International Conference RANLP 2009 - Borovets, Bulgaria, pages 251–257 Diacritization for Real-World Arabic Texts Emad Mohamed Indiana University emohamed@indiana.edu Sandra K¨ubler Indiana University skuebler@indiana.edu Abstract For Arabic, diacritizing written text is impor- tant for many NLP tasks. In the work presented here, we investigate the quality of a diacritization approach, with a high success rate for treebank data but with a more limited success on real- world data. One of the problems we encountered is the non-standard use of the hamza diacritic, which leads to a decrease in diacritization accu- racy. If an automatic hamza restoration mod- ule precedes diacritization, the results improve from a word error rate of 9.20% to 7.38% in tree- bank data, and from 7.96% to 5.93% on selected real-world texts. This shows clearly that hamza restoration is a necessary step for improving di- acritization quality for Arabic real-world texts. Keywords Arabic diacritization, memory-based learning 1 Introduction The problem of diacritization, or vocalization, is es- sential to many tasks in Arabic NLP. Arabic is gen- erally written without the short vowels, which means that one written form can have several pronunciations, each pronunciation carrying its own meaning(s). The word form ’mskn’ is an example for a highly ambigu- ous word. Its possible pronunciations include ’maskan’ (home), ’musakkin’ (analgesic), ’masakn’ (they-fem. have held), or ’musikn’ (they-fem. have been held). The importance of diacritization becomes clear when we look at how Google Translate renders ’A$tryt Alm- skn mn AlSydlyp’ (I bought a pain killer from the pharmacy): as ’I bought the home from the phar- macy’. This error would not occur if the input to the translation system were diacritized in a first step be- fore the actual translation process. However, diacriti- zation is far from trivial: the example above shows that the diacritized forms of a single undiacritized word differ in their parts of speech (POS) as well as in their meaning. This shows that to a certain degree, diacritization performs implicit POS tagging and word sense disambiguation. It also shows very clearly that word forms cannot be diacritized in isolation; the task is heavily dependent on the context of the word. The goal of the work reported here is the creation of a diacritization system that works reliably on real- world data taken from websites. The first step towards this problem is a system that reaches a high accuracy in an in-vitro evaluation, i.e. on data from the same treebank on which it was trained. After we obtain a reliable system for the treebank data, we will test it to determine how well it works in an in-vivo evaluation, i.e. on real-world data. More importantly, in the sec- ond step, we will investigate how the system needs to be modified in order to produce reliable diacritizations in the in-vivo situation. In the first step, we concentrate on investigating how important different types of context are for diacritiza- tion. We follow Zitouni et al. [16] in defining diacriti- zation as a classification problem in which we decide …

Content truncated to 500 words