Creating the best development corpus for Statistical Machine Translation systems

Creating the best development corpus for Statistical Machine Translation systems Mara Chinea-Rios1 Germ´an Sanchis-Trilles2 1 Pattern Recognition and Human Language Technology Research Center Universitat Polit`ecnica de Val`encia, Val`encia, Spain {machirio, fcn}@prhlt.upv.es 2 Sciling, Val`encia, Spain sanchis@sciling.es Francisco Casacuberta1 Abstract We propose and study three different novel approaches for tackling the problem of de- velopment set selection in Statistical Ma- chine Translation. We focus on a sce- nario where a machine translation sys- tem is leveraged for translating a specific test set, without further data from the do- main at hand. Such test set stems from a real application of machine translation, where the texts of a specific e-commerce were to be translated. For developing our development-set selection techniques, we first conducted experiments in a controlled scenario, where labelled data from differ- ent domains was available, and evaluated the techniques both with classification and translation quality metrics. Then, the best- performing techniques were evaluated on the e-commerce data at hand, yielding con- sistent improvements across two language directions. 1 Introduction Tuning is a critical step in every system that presents a weighted combination of features. By adjusting the weights so that they best fit the target distribution, this process typically yields important improvements on the performance of the system developed. However, selecting an appropriate de- velopment set is key for this process to reach its goal. In Statistical Machine Translation (SMT), the tuning step implies optimizing the log-linear c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. weights {λ1 . . . λm . . . λM} of a discriminative model that implements a weighted combination of features {h1 . . . hm . . . hM}, considered relevant in the translation process: ˆy = argmax y M � m=1 λmhm(x, y) (1) with x and y being the source and target sentences. Such optimization has become de-facto standard in SMT, thanks to the wide-spread adoption of tun- ing algorithms such as Minimum Error Rate Train- ing (MERT) (Och, 2003) or the Margin Infused Relaxed Algorithm (MIRA) (Cherry and Foster, 2012). The purpose of these algorithms is to adjust the log-linear weights such that the model distri- bution best fits the target distribution, or the target metric by which the system is evaluated. Given that the amount of weights λm is typically around 10 or 20, the size of the development cor- pus required for tuning is typically in the range of hundreds or a few thousands of sentences. How- ever, such corpus is typically required to be dis- joint from the training corpus, used to estimate the features hm, and its selection is critical, having an important impact on the system’s performance if the development set of choice is too different from the test set at hand (Koehn, 2010). The Data Selection (DS) task is stated as the problem of selecting the best sub-corpus of sen- tences from an available pool of sentences, with which to train a machine …

Content truncated to 500 words