Investigating the Impact of Various Partial Diacritization Schemes on Arabic-English Statistical Machine Translation

Investigating the Impact of Various Partial Diacritization Schemes on Arabic-English Statistical Machine Translation Sawsan Alqahtani, Mahmoud Ghoneim, Mona Diab {sawsanq, mghoneim, mtdiab}@gwu.edu Department of Computer Science The George Washington University, USA Washington, DC 20052 Abstract Most diacritics in Arabic represent short vowels. In Arabic orthography, such diacritics are considered optional. The absence of these diacritics naturally leads to significant word am- biguity to top the inherent ambiguity present in fully diacritized words. Word ambiguity is a significant impediment for machine translation. Despite the ambiguity presented by lack of diacritization, context helps ameliorate the situation. Identifying the appropriate amount of di- acritic restoration to reduce word sense ambiguity in the context of machine translation is the object of this paper. Diacritic marks help reduce the number of possible lexical word choices assigned to a source word which leads to better quality translated sentences. We investigate a variety of (linguistically motivated) partial diacritization schemes that preserve some of the semantics that in essence complement the implicit contextual information present in the sen- tences. We also study the effect of training data size and report results on three standard test sets that represent a combination of different genres. The results show statistically significant improvements for some schemes compared to two baselines: text with no diacritics (the typical writing system adopted for Arabic) and text that is fully diacritized. 1 Introduction Resolving natural language ambiguity is at the crux of the NLP enterprise. Ambiguity refers to the problem of possibly having different interpretations for different segments (words, phrases, etc.) of a sentence. Languages such as Arabic, Hebrew and Persian are typically written in a manner that exacerbates this ambiguity problem and increases the homograph rate by under- specifying some of the letters such as short vowels and consonantal gemination, which in turn increases the effect of having multiple interpretations for the same word. This renders text even more ambiguous than typically expected. While context helps native speakers of the language resolve some of the ambiguity, con- text alone does not always produce adequate clarity for interpretation. The problem is further complicated in Arabic by the fact that there are no native speakers of Modern Standard Ara- bic (MSA), which is the language used in education and formal settings. Instead, speakers of Arabic converse in various dialects of Arabic which are at times starkly different from MSA. One solution for this problem is diacritic restoration, or diacritization, which refers to ren- ������������������������������������������������������ ����������������������������������������� dering the underspecified diacritics explicit in the text. We investigate the problem of diacritiza- tion within the context of Arabic-to-English Statistical Machine Translation (SMT) system. We address the problem in MSA texts, the majority of which are underspecified for these diacritic marks. We focus here on the most prominent Arabic diacritics which are short vowels (i, u, a), the syllable boundary marker, known as sukoon (o), indefiniteness marker, known as nunation (F, K, N), and the consonantal doubling marker (gemination) known as shadda (∼)1. In this study, we aim to investigate what …

Content truncated to 500 words