Orthographic and Morphological Processing for Persian-to-English Statistical Machine Translation

International Joint Conference on Natural Language Processing, pages 1047–1051, Nagoya, Japan, 14-18 October 2013. Orthographic and Morphological Processing for Persian-to-English Statistical Machine Translation Mohammad Sadegh Rasooli and Ahmed El Kholy and Nizar Habash Center for Computational Learning Systems Columbia University, New York, NY {rasooli,akholy,habash}@ccls.columbia.edu Abstract In statistical machine translation, data sparsity is a challenging problem espe- cially for languages with rich morphology and inconsistent orthography, such as Per- sian. We show that orthographic prepro- cessing and morphological segmentation of Persian verbs in particular improves the translation quality of Persian-English by 1.9 BLEU points on a blind test set. 1 Introduction In the context of statistical machine translation (SMT), the severity of the data sparsity problem, typically a result of limited parallel data, increases for languages with rich morphology such as Ara- bic, Czech and Turkish. The most common solu- tion, other than increasing the amount of parallel data, is to develop language-specific preprocess- ing and tokenization schemes that reduce the over- all vocabulary and increase the symmetry between source and target languages (Nießen and Ney, 2004; Lee, 2004; Oflazer and Durgar El-Kahlout, 2007; Stymne, 2012; Singh and Habash, 2012; Habash and Sadat, 2012; El Kholy and Habash, 2012). In this paper, we work with Persian, a mor- phologically rich language with limited parallel data. Furthermore, Persian’s standard orthography makes use of a combination of spaces and semi- spaces (zero-width non-joiners), which are often ignored or confused, leading to orthographic in- constancies and added sparsity. We address the or- thographic challenge of inconsistent spacing with a supervised learning method which successfully recovers near all spacing errors. We also present a set of experiments for morphological segmenta- tion to help improve Persian-to-English SMT. We show that the combination of orthographic cleanup and morphological segmentation for verbs in par- ticular improves over a simple preprocessing base- line. 2 Related Work Much work has been done to address data spar- sity in SMT employing a variety of methods such as morphological and orthographic processing (Nießen and Ney, 2004; Lee, 2004; Goldwater and McClosky, 2005; Oflazer and Durgar El-Kahlout, 2007; Stymne, 2012; Singh and Habash, 2012; Habash and Sadat, 2012; El Kholy and Habash, 2012), targeting specific out-of-vocabulary phe- nomena with name transliteration or spelling ex- pansion (Habash, 2008; Hermjakob et al., 2008) or using comparable corpora (Prochasson and Fung, 2011). Our approach falls in the class of ortho- graphic and morphological preprocessing. Previous research on Persian SMT is rather lim- ited despite some early efforts (Amtrup et al., 2000). A few parallel corpora have been released, such as (Pilevar et al., 2011; Farajian, 2011). We conduct our research on an unreleased Persian- English parallel corpus (El Kholy et al., 2013a; El Kholy et al., 2013b). In terms of preprocessing efforts, Kathol and Zheng (2008) use unsupervised Persian morpheme segmentation. Other attempts to improve Persian SMT use syntactic reordering (Gupta et al., 2012; Matusov and K¨opr¨u, 2010) and rule-based post editing (Mohaghegh et al., 2012). El Kholy et al. (2013a) and El Kholy et al. …

Content truncated to 500 words