Exploring Consensus in Machine Translation for Quality Estimation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 342–347, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Exploring Consensus in Machine Translation for Quality Estimation Carolina Scarton and Lucia Specia Department of Computer Science, University of Sheffield Regent Court, 211 Portobello, Sheffield, S1 4DP, UK {c.scarton,l.specia}@sheffield.ac.uk Abstract This paper presents the use of consensus among Machine Translation (MT) systems for the WMT14 Quality Estimation shared task. Consensus is explored here by com- paring the MT system output against sev- eral alternative machine translations using standard evaluation metrics. Figures ex- tracted from such metrics are used as fea- tures to complement baseline prediction models. The hypothesis is that knowing whether the translation of interest is simi- lar or dissimilar to translations from multi- ple different MT systems can provide use- ful information regarding the quality of such a translation. 1 Introduction While Machine Translation (MT) evaluation met- rics can rely on the similarity of the MT system output to reference (human) translations as a proxy to quality assessment, this is not possible for MT systems in use, translating unseen texts. Quality Estimation (QE) metrics are used in such settings as a way of predicting translation quality. While reference translations are not available for QE, previous work has explored the so called pseudo- references (Soricut and Echihabi, 2010; Soricut et al., 2012; Soricut and Narsale, 2012; Shah et al., 2013). Pseudo-references are alternative transla- tions produced by MT systems different from the system that we intend to predict quality for (Al- brecht and Hwa, 2008). These can be used to pro- vide additional features to train QE models. Such features are normally figures resulting from au- tomatic metrics (such as BLEU, Papineni et al. (2002)) computed between pseudo-references and the output of the given MT system. Soricut and Echihabi (2010) explore pseudo- references for document-level QE prediction to rank outputs from an MT system. The pseudo- references-based features are BLEU scores ex- tracted by comparing the output of the MT sys- tem under investigation and the output of an off- the-shelf MT system, for both the target and the source texts. The statistical MT system training data is also used as pseudo-references to compute training data-based features. The use of pseudo- references has been shown to outperform strong baseline results. Soricut and Narsale (2012) pro- pose a method that uses sentence-level prediction models for document-level QE. They also use a pseudo-references-based feature (based in BLEU) and claim that this feature is one of the most pow- erful in the framework. For QE at sentence-level, Soricut et al. (2012) use BLEU based on pseudo-references combined with other features to build the best QE system of the WMT12 QE shared task.1 Shah et al. (2013) use pseudo-references in the same way to ex- tract a BLEU feature for sentence-level prediction. Feature analysis on a number of datasets showed that this feature contributed the most across all datasets. Louis and Nenkova (2013) apply pseudo- references for summary evaluation. They use …

Content truncated to 500 words