UPF-Cobalt Submission to WMT15 Metrics Task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 373–379, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. UPF-Cobalt Submission to WMT15 Metrics Task Marina Fomicheva N´uria Bel Iria da Cunha IULA, Universitat Pompeu Fabra {marina.fomicheva,nuria.bel,iria.dacunha}@upf.edu Anton Malinovskiy Nuroa Internet S. L. amalinovskiy@gmail.com Abstract An important limitation of automatic eval- uation metrics is that, when comparing Machine Translation (MT) to a human ref- erence, they are often unable to discrimi- nate between acceptable variation and the differences that are indicative of MT er- rors. In this paper we present UPF-Cobalt evaluation system that addresses this issue by penalizing the differences in the syntac- tic contexts of aligned candidate and refer- ence words. We evaluate our metric using the data from WMT workshops of the re- cent years and show that it performs com- petitively both at segment and at system levels. 1 Introduction Current automatic MT evaluation methods are grounded on the following key idea: the closer an MT is to a professional Human Translation (HT), the higher its quality. Thus, metrics typ- ically calculate evaluation scores based on some sort of similarity between machine and human translations. The performance of evaluation sys- tems is in its turn evaluated by calculating the correlation with human judgments. Manual qual- ity assessment can be conducted in various ways: adequacy and fluency scoring, calculating post- editing cost or post-editing time, error analysis, ranking, etc. In the latter case, humans are asked to compare the outputs of different MT systems and rank them in terms of quality. Ranking- based evaluation has gained a lot of attention in the recent years and is used in important evalu- ation campaigns such as the Metrics task at the Workshop on Machine Translation (WMT). This This work was supported by IULA (UPF) and the FI- DGR grant program of the Generalitat de Catalunya. setting is preferred, since it has been shown to yield higher inter-annotator agreement than ab- solute quality assessment (Callison-Burch et al., 2007). In our opinion, one of the main reasons why the correlation between automatic evaluation and hu- man rankings is still not satisfactory is that met- rics’ scores are not discriminative enough to ap- proximate human comparisons. Given various candidate translations of the same source sentence, all of them different from the reference, evaluation systems are often unable to determine which trans- lation is better as they cannot tell apart candidate- reference differences related to acceptable linguis- tic variation and the differences induced by MT errors. Furthermore, if all candidate translations contain a number of translation errors, metrics fail to predict the human ranking because they make no estimation of the relative importance of differ- ent types of MT errors for the overall translation quality. We suggest that the aforementioned limitations can be addressed by means of enhancing word comparison with contextual information. Varia- tion between two translation options is acceptable if semantically similar words in the corresponding sentences occur in equivalent contexts. In case of translation errors either the lexical …

Content truncated to 500 words