Uzbek-English and Turkish-English Morpheme Alignment Corpora

Uzbek-English and Turkish-English Morpheme Alignment Corpora Xuansong Li, Jennifer Tracey, Stephen Grimes, Stephanie Strassel Linguistic Data Consortium, University of Pennsylvania Philadelphia, PA 19104 USA Email:{xuansong,garjen,sgrimes,strassel}@ldc.upenn.edu Abstract Morphologically-rich languages pose problems for machine translation (MT) systems, including word-alignment errors, data sparsity and multiple affixes. Current alignment models at word-level do not distinguish words and morphemes, thus yielding low-quality alignment and subsequently affecting end translation quality. Models using morpheme-level alignment can reduce the vocabulary size of morphologically-rich languages and overcomes data sparsity. The alignment data based on smallest units reveals subtle language features and enhances translation quality. Recent research proves such morpheme-level alignment (MA) data to be valuable linguistic resources for SMT, particularly for languages with rich morphology. In support of this research trend, the Linguistic Data Consortium (LDC) created Uzbek-English and Turkish-English alignment data which are manually aligned at the morpheme level. This paper describes the creation of MA corpora, including alignment and tagging process and approaches, highlighting annotation challenges and specific features of languages with rich morphology. The light tagging annotation on the alignment layer adds extra value to the MA data, facilitating users in flexibly tailoring the data for various MT model training. Keywords: machine translation, morpheme alignment, word alignment 1. Introduction MT alignment has been an active research area for enhancing MT quality. Modern statistical machine translation (SMT) systems typically use “word” as the atomic unit for translation. Word-alignment is the initial step in SMT pipeline, aiming to identify word correspondence of source and target languages. As current word alignment models do not address morphology below the word level, it can be difficult to divide the MT problem into sub-problems and tackle each sub-problem in isolation to improve the overall quality of MT. The problem is particularly outstanding for morphologically rich languages as statistical correspondences between source and target words are diffused over many morphological forms (multiple surface forms for a morpheme). The fact that rare words and multiple affixes often occur in highly inflected languages exacerbates this problem. Morpheme- based alignment is useful in the translation process of highly inflected languages. Morphological inflections indicate tense, gender or number which are normally expressed as separate words in uninflected languages. Capturing such sub-word alignments can yield better word alignments. Recent SMT research has found that utilizing information from morphology improves the quality of word alignments. Eyig¨oz et al. (2013) developed a MT model using a two-level Turkish-English alignment, achieving significant improvement of BLEU scores. Luong and Kan (2010) proposed a morphologically sensitive approach to word alignment for language pairs involving a highly inflected language, addressing morpheme alignment issues which are peculiar to highly inflected languages. Toutanova et al. (2008) improved the quality of SMT by applying inflection generation models that predict word forms from their stems using extensive morphological and syntactic information from both the source and target languages, Russian and Arabic. Their model improves the quality of SMT over both phrasal and syntax-based approaches. Minkov et al. (2007) adopted a novel method for predicting inflected word forms for …

Content truncated to 500 words