Exploring Prediction Uncertainty in Machine Translation Quality Estimation

Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 208–218, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Exploring Prediction Uncertainty in Machine Translation Quality Estimation Daniel Beck† Lucia Specia† Trevor Cohn‡ †Department of Computer Science University of Sheffield, United Kingdom ‡Computing and Information Systems University of Melbourne, Australia {debeck1,l.specia}@sheffield.ac.uk, t.cohn@unimelb.edu.au Abstract Machine Translation Quality Estimation is a notoriously difficult task, which lessens its usefulness in real-world translation en- vironments. Such scenarios can be im- proved if quality predictions are accompa- nied by a measure of uncertainty. How- ever, models in this task are tradition- ally evaluated only in terms of point es- timate metrics, which do not take predic- tion uncertainty into account. We investi- gate probabilistic methods for Quality Es- timation that can provide well-calibrated uncertainty estimates and evaluate them in terms of their full posterior predictive dis- tributions. We also show how this poste- rior information can be useful in an asym- metric risk scenario, which aims to capture typical situations in translation workflows. 1 Introduction Quality Estimation (QE) (Blatz et al., 2004; Spe- cia et al., 2009) models aim at predicting the quality of automatically translated text segments. Traditionally, these models provide point esti- mates and are evaluated using metrics like Mean Absolute Error (MAE), Root-Mean-Square Error (RMSE) and Pearson’s r correlation coefficient. However, in practice QE models are built for use in decision making in large workflows involving Machine Translation (MT). In these settings, rely- ing on point estimates would mean that only very accurate prediction models can be useful in prac- tice. A way to improve decision making based on quality predictions is to explore uncertainty esti- mates. Consider for example a post-editing sce- nario where professional translators use MT in an effort to speed-up the translation process. A QE model can be used to determine if an MT seg- ment is good enough for post-editing or should be discarded and translated from scratch. But since QE models are not perfect they can end up al- lowing bad MT segments to go through for post- editing because of a prediction error. In such a sce- nario, having an uncertainty estimate for the pre- diction can provide additional information for the filtering decision. For instance, in order to ensure good user experience for the human translator and maximise translation productivity, an MT segment could be forwarded for post-editing only if a QE model assigns a high quality score with low uncer- tainty (high confidence). Such a decision process is not possible with point estimates only. Good uncertainty estimates can be acquired from well-calibrated probability distributions over the quality predictions. In QE, arguably the most successful probabilistic models are Gaussian Pro- cesses (GPs) since they considered the state-of- the-art for regression (Cohn and Specia, 2013; Hensman et al., 2013), especially in the low-data regimes typical for this task. We focus our anal- ysis in this paper on GPs since other common models used in QE can only provide point esti- …

Content truncated to 500 words