Analysing the Effect of Out-of-Domain Data on SMT Systems

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 422–432, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Analysing the Effect of Out-of-Domain Data on SMT Systems Barry Haddow and Philipp Koehn School of Informatics University of Edinburgh Edinburgh, EH8 9AB, Scotland {bhaddow,pkoehn}@inf.ed.ac.uk Abstract In statistical machine translation (SMT), it is known that performance declines when the training data is in a different domain from the test data. Nevertheless, it is frequently nec- essary to supplement scarce in-domain train- ing data with out-of-domain data. In this pa- per, we first try to relate the effect of the out- of-domain data on translation performance to measures of corpus similarity, then we sep- arately analyse the effect of adding the out- of-domain data at different parts of the train- ing pipeline (alignment, phrase extraction, and phrase scoring). Through experiments in 2 do- mains and 8 language pairs it is shown that the out-of-domain data improves coverage and translation of rare words, but may degrade the translation quality for more common words. 1 Introduction In statistical machine translation (SMT), domain adaptation can be thought of as the problem of train- ing a system on data mainly drawn from one do- main (e.g. parliamentary proceedings) and trying to maximise its performance on a different domain (e.g. news). There is likely to be some parallel data similar to the test data, but as such data is expen- sive to create, it tends to be scarce. The concept of “domain” is rarely given a precise definition, but it is normally understood that data from the same domain is in some sense similar (for example in the words and grammatical constructions used) and data from different domains shows less similarities. Data from the same domain as the test set is usually referred to as in-domain and data from a different domain is referred to as out-of-domain. The aim of this paper is to shed some light on what domain actually is, and why it matters. The fact that a mismatch between training and test data domains reduces translation performance has been observed in previous studies, and will be confirmed here for multiple data sets and languages, but the question of why domain matters for performance has not been fully addressed in the literature. Exper- iments in this paper will be conducted on phrase- based machine translation (PBMT) systems, but similar conclusions are likely to apply to other types of SMT systems. Furthermore, we will mainly be concerned with the effect of domain on the transla- tion model, since it depends on parallel data which is more likely to be in short supply than monolingual data, and domain adaptation for language modelling has been more thoroughly studied. The effect of a shift of domain in the parallel data is complicated by the fact that training a translation model is a multi-stage process. First the parallel data is word-aligned, normally using the IBM mod- els (Brown et al., 1994), then phrases are extracted using some heuristics …

Content truncated to 500 words