Learning Machine Translation from In-domain and Out-of-domain Data

Learning Machine Translation from In-domain and Out-of-domain Data Marco Turchi European Commission JRC, IPSC - GlobeSec Via Fermi 2749, 21020 Ispra (VA), Italy marco.turchi@jrc.ec.europa.eu Cyril Goutte Interactive Language Tech., National Research Council Canada, 283 Boulevard Alexandre-Tach´e, Gatineau QC J8X3X7, Canada Cyril.Goutte@nrc.ca Nello Cristianini Intelligent Systems Lab., University of Bristol, MVB, Woodland Rd, BS8-1 UB, Bristol, UK Nello@support-vector.net Abstract The performance of Phrase-Based Statis- tical Machine Translation (PBSMT) sys- tems mostly depends on training data. Many papers have investigated how to cre- ate new resources in order to increase the size of the training corpus in an at- tempt to improve PBSMT performance. In this work, we analyse and characterize the way in which the in-domain and out- of-domain performance of PBSMT is im- pacted when the amount of training data increases. Two different PBSMT systems, Moses and Portage, two of the largest par- allel corpora, Giga (French-English) and UN (Chinese-English) datasets and several in- and out-of-domain test sets were used to build high quality learning curves show- ing consistent logarithmic growth in per- formance. These results are stable across language pairs, PBSMT systems and do- mains. We also analyse the respective im- pact of additional training data for esti- mating the language and translation mod- els. Our proposed model approximates learning curves very well and indicates the translation model contributes about 30% more to the performance gain than the lan- guage model. 1 Introduction With the growing availability of bilingual parallel corpora, the past two decades saw the development and widespread adoption of statistical machine translation (SMT) models. Given a source (“for- eign”) language sentence f and a target (“english”) c⃝ 2012 European Association for Machine Translation. language translation e, the relationship between e and f is modelled using a statistical or probabilis- tic model which is estimated from a large amount of textual data, comprising bilingual and monolin- gual corpora. The most popular class of SMT sys- tems is Phrase-Based SMT (PBSMT, (Koehn et al., 2003)). In this paper, we are concerned with analyzing and characterizing the way in which the perfor- mance of PBSMT models evolves with increasing amounts of training data. In the SMT community, it is a common belief that learning curves follow logarithmic laws. However, there are few large- scale systematic analyses of the growth rate of the PBSMT performance. Early work (Al-Onaizan et al., 1999) used a relatively small training set and perplexity as evaluation metric. (Koehn et al., 2003) and (Suresh, 2010) show that BLEU score has a log-linear dependency with training corpus size, but this is limited to 350k training sentence pairs. Learning curves were also pre- sented in order to motivate the use of active learn- ing for MT (Bloodgood and Callison-Burch, 2010; Haffari et al., 2011). They attempt to address the challenge of “diminishing returns” in learn- ing MT, although this is again done with small training corpora (<90k sentence pairs), and, on a log-scale, performance seems again to increase lin- early. (Brants et al., 2007) produced a large-scale study, …

Content truncated to 500 words