Predicting Success in Machine Translation

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745–754, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Predicting Success in Machine Translation Alexandra Birch Miles Osborne a.c.birch-mayne@sms.ed.ac.uk miles@inf.ed.ac.ukPhilipp Koehn pkoehn@inf.ed.ac.uk School of Informatics University of Edinburgh 10 Crichton Street Edinburgh, EH8 9AB, UK Abstract The performance of machine translation sys- tems varies greatly depending on the source and target languages involved. Determining the contribution of different characteristics of language pairs on system performance is key to knowing what aspects of machine transla- tion to improve and which are irrelevant. This paper investigates the effect of different ex- planatory variables on the performance of a phrase-based system for 110 European lan- guage pairs. We show that three factors are strong predictors of performance in isolation: the amount of reordering, the morphological complexity of the target language and the his- torical relatedness of the two languages. To- gether, these factors contribute 75% to the variability of the performance of the system. 1 Introduction Statistical machine translation (SMT) has improved over the last decade of intensive research, but for some language pairs, translation quality is still low. Certain systematic differences between languages can be used to predict this. Many researchers have speculated on the reasons why machine translation is hard. However, there has never been, to our knowl- edge, an analysis of what the actual contribution of different aspects of language pairs is to translation performance. This understanding of where the diffi- culties lie will allow researchers to know where to most gainfully direct their efforts to improving the current models of machine translation. Many of the challenges of SMT were first out- lined by Brown et al. (1993). The original IBM Models were broken down into separate translation and distortion models, recognizing the importance of word order differences in modeling translation. Brown et al. also highlighted the importance of mod- eling morphology, both for reducing sparse counts and improving parameter estimation and for the cor- rect production of translated forms. We see these two factors, reordering and morphology, as fundamental to the quality of machine translation output, and we would like to quantify their impact on system per- formance. It is not sufficient, however, to analyze the mor- phological complexity of the source and target lan- guages. It is also very important to know how sim- ilar the morphology is between the two languages, as two languages which are morphologically com- plex in very similar ways, could be relatively easy to translate. Therefore, we also include a measure of the family relatedness of languages in our analysis. The impact of these factors on translation is mea- sured by using linear regression models. We perform the analysis with data from 110 different language pairs drawn from the Europarl project (Koehn, 2005). This contains parallel data for the 11 official language pairs of the European Union, providing a rich variety of different language characteristics for our experiments. Many research papers report re- sults on only one or two languages …

Content truncated to 500 words