Sentence-level quality estimation by predicting HTER as a multi-component metric

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 534–539 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Sentence-level quality estimation by predicting HTER as a multi-component metric Eleftherios Avramidis German Research Center for Artificial Intelligence (DFKI) Language Technology Lab, Berlin, Germany eleftherios.avramidis@dfki.de Abstract This submission investigates alternative machine learning models for predicting the HTER score on the sentence level. Instead of directly predicting the HTER score, we suggest a model that jointly pre- dicts the amount of the 4 distinct post- editing operations, which are then used to calculate the HTER score. This also gives the possibility to correct invalid (e.g. neg- ative) predicted values prior to the calcula- tion of the HTER score. Without any fea- ture exploration, a multi-layer perceptron with 4 outputs yields small but significant improvements over the baseline. 1 Introduction Quality Estimation (QE) is the evaluation method that aims at employing machine learning in order to predict some measure of quality given a Ma- chine Translation (MT) output (Blatz et al., 2004). A commonly-used subtask of QE refers to the learning of automatic metrics. These metrics pro- duce a continuous score based on the comparison between the MT output and a reference transla- tion. When the reference is a minimal post-edition of the MT output, the quality score produced is in- tuitively more objective and robust as compared to other QE subtasks, where the quality score is as- signed directly by the annotators. In that case, the score is a direct reflection of the changes that need to take place in order to fix the translation. HTER (Snover et al., 2009) is the most commonly used metric as it directly represents the least required post-editing effort. In order to predict the results of an automatic metric, QE approaches use machine learning to predict a model that associates a feature vector with the single quality score. In this case the statis- tical model treats the automatic metric as a black box, in the sense that no particular knowledge about the exact calculation of the quality score is explicitly included in the model. In this submission we aim to partially break this black-box. We explore the idea of creating a QE model that does not directly predict the single HTER score, but it jointly predicts the 4 compo- nents of the metric, which are later used for com- puting the single score. This way, the structure of the model can be aware of the distinct factors that comprise the final quality score and also poten- tially learn the interactions between them. Hence, the focus of this submission will remain on ma- chine learning and there will not be exploration in terms of features. In Section 2 we briefly in- troduce previous work, in Section 3 we provide details about the method, whereas the experiment results are given in Section 4. In Section 5 we de- scribe the models submitted at the shared-task and we explain why they differ from …

Content truncated to 500 words