Maximum Entropy Based Restoration of Arabic Diacritics

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 577–584, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Maximum Entropy Based Restoration of Arabic Diacritics Imed Zitouni, Jeffrey S. Sorensen, Ruhi Sarikaya IBM T.J. Watson Research Center 1101 Kitchawan Rd, Yorktown Heights, NY 10598 {izitouni, sorenj, sarikaya}@us.ibm.com Abstract Short vowels and other diacritics are not part of written Arabic scripts. Exceptions are made for important political and reli- gious texts and in scripts for beginning stu- dents of Arabic. Script without diacritics have considerable ambiguity because many words with different diacritic patterns ap- pear identical in a diacritic-less setting. We propose in this paper a maximum entropy approach for restoring diacritics in a doc- ument. The approach can easily integrate and make effective use of diverse types of information; the model we propose inte- grates a wide array of lexical, segment- based and part-of-speech tag features. The combination of these feature types leads to a state-of-the-art diacritization model. Using a publicly available corpus (LDC’s Arabic Treebank Part 3), we achieve a di- acritic error rate of 5.1%, a segment error rate 8.5%, and a word error rate of 17.3%. In case-ending-less setting, we obtain a di- acritic error rate of 2.2%, a segment error rate 4.0%, and a word error rate of 7.2%. 1 Introduction Modern Arabic written texts are composed of scripts without short vowels and other diacritic marks. This often leads to considerable ambigu- ity since several words that have different diacritic patterns may appear identical in a diacritic-less setting. Educated modern Arabic speakers are able to accurately restore diacritics in a document. This is based on the context and their knowledge of the grammar and the lexicon of Arabic. However, a text without diacritics becomes a source of confu- sion for beginning readers and people with learning disabilities. A text without diacritics is also prob- lematic for applications such as text-to-speech or speech-to-text, where the lack of diacritics adds another layer of ambiguity when processing the data. As an example, full vocalization of text is required for text-to-speech applications, where the mapping from graphemes to phonemes is simple compared to languages such as English and French; where there is, in most cases, one-to-one relation- ship. Also, using data with diacritics shows an improvement in the accuracy of speech-recognition applications (Afify et al., 2004). Currently, text-to- speech, speech-to-text, and other applications use data where diacritics are placed manually, which is a tedious and time consuming excercise. A di- acritization system that restores the diacritics of scripts, i.e. supply the full diacritical markings, would be of interest to these applications. It also would greatly benefit nonnative speakers, sufferers of dyslexia and could assist in restoring diacritics of children’s and poetry books, a task that is cur- rently done manually. We propose in this paper a statistical approach that restores diacritics in a text document. The proposed approach is based on the maximum en- tropy framework where several diverse sources of information …

Content truncated to 500 words