Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2205–2216, Dublin, Ireland, August 23-29 2014. Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem Erwan Moreau CNGL and Computational Linguistics Group Centre for Computing and Language Studies School of Computer Science and Statistics Trinity College Dublin Dublin 2, Ireland moreaue@cs.tcd.ie Carl Vogel Computational Linguistics Group Centre for Computing and Language Studies School of Computer Science and Statistics Trinity College Dublin Dublin 2, Ireland vogel@cs.tcd.ie Abstract In this paper we address the question of the reliability of the predictions made by MT Quality Estimation (QE) systems. In particular, we show that standard supervised QE systems, usually trained to minimize MAE, make serious mistakes at predicting the quality of the sentences in the tails of the quality range. We describe the problem and propose several experiments to clarify their causes and effects. We use the WMT12 and WMT13 QE Shared Task datasets to prove that our claims hold in general and are not specific to a dataset or a system. 1 Introduction Machine Translation (MT) Quality Estimation (QE) has become an important subject of study in the past few years (Callison-Burch et al., 2012; Bojar et al., 2013). This follows directly from the erratic quality of MT output in general: although MT is now widely used in professional contexts, it is still prone to many errors; therefore a careful post-editing stage, performed by human experts, is usually needed. In this context, QE can help carrying out this process more efficiently, and more specifically to help in the decision process between the automatic and the manual stages: if a reliable indication of quality is provided for every machine-translated sentence, the human effort can be reduced. For example, a very bad translation is worthless because the translator usually has to spend more time fixing it than she or he would have spent translating the sentence from scratch; thus it makes more sense in such cases to either send the sentence back to an alternative MT system (e.g. trained on a different corpus), or simply leave it untranslated for the translator. Clearly the advantage of using a QE system depends on the reliability of its predictions. If it makes too many errors, then it only confuses the translation workflow; in this case the translators would perform better without it. The quality of an (automatic) QE system cannot be perfect, but it should be at least controllable. That is, it should be possible to assess the reliability of the predictions made by a system, for instance by estimating the level of confidence of the predictions. Hopefully, QE systems will progress towards this kind of behaviour, but currently the evaluation methods are not entirely satisfactory from this perspective. In particular, after describing our experimental setting in §2, we will observe in §3 that the use of the Mean Absolute Error1 (MAE) as a global evaluation measure hides huge discrepancies in the distribution of errors among the range of scores. More …

Content truncated to 500 words