Results of the WMT15 Tuning Shared Task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 274–281, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Results of the WMT15 Tuning Shared Task Miloˇs Stanojevi´c and Amir Kamran University of Amsterdam ILLC {m.stanojevic,a.kamran}@uva.nl Ondˇrej Bojar Charles University in Prague MFF ´UFAL bojar@ufal.mff.cuni.cz Abstract This paper presents the results of the WMT15 Tuning Shared Task. We pro- vided the participants of this task with a complete machine translation system and asked them to tune its internal parameters (feature weights). The tuned systems were used to translate the test set and the out- puts were manually ranked for translation quality. We received 4 submissions in the English-Czech and 6 in the Czech-English translation direction. In addition, we ran 3 baseline setups, tuning the parameters with standard optimizers for BLEU score. 1 Introduction Almost all modern statistical machine translation (SMT) systems internally consider translation can- didates from several aspects. Some of these as- pects can be very simple and one parameter is suf- ficient to capture them, such as the word penalty incurred for every word produced or the phrase penalty controlling whether the sentence should be translated in fewer or more independent phrases, leading to more or less word-for-word translation. Other aspects try to assess e.g. the fidelity of the translation, the fluency of the output or the amount of reordering. These are far more complex and for- mally captured in a model such as the translation model or language model. Both the simple penalties as well as the scores from the more complex models are called features and need to be combined to a single score to allow for ranking of translation candidates. This is usu- ally done using a linear combination of the scores: score(e) = M ∑ m=1 λmhm(e, f) (1) where e and f are the candidate translation and the source, respectively, and hm(·, ·) is one of the M penalties or models. The tuned parameters are λm ∈ R, called feature weights. Feature weights have a tremendous effect on the final translation quality. For instance the sys- tem can produce extremely long outputs, fabulat- ing words just in order to satisfy a negatively- weighted word penalty, i.e. a bonus for each word produced. An inherent part of the preparation of MT systems is thus some optimization of the weight settings. If we had to set the weights manually, we would have to try a few configurations and pick one that leads to reasonable outputs. The common prac- tice is to use an optimization algorithm that ex- amines many settings, evaluating the produced translations automatically against reference trans- lations using some evaluation measure (tradition- ally called “metric” in the MT field). In short, the optimizer tunes model weights so that the final combined model score correlates with the metric score. The metric score, in turn, is designed to cor- relate well with human judgements of translation quality, see Stanojevi´c et al. (2015) and the pre- vious papers summarizing WMT metrics tasks. However, a …

Content truncated to 500 words