Non-Linear Models for Confidence Estimation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 157–162, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Non-Linear Models for Confidence Estimation Yong Zhuang∗ Zhejiang University 866 Yuhangtang Road Hangzhou, China yong.zhuang22@gmail.com Guillaume Wisniewski and Fran¸cois Yvon Univ. Paris Sud and LIMSI–CNRS rue John von Neumann 91403 Orsay CEDEX, France {firstname.lastname}@limsi.fr Abstract This paper describes our work with the data distributed for the WMT’12 Confidence Es- timation shared task. Our contribution is twofold: i) we first present an analysis of the data which highlights the difficulty of the task and motivates our approach; ii) we show that using non-linear models, namely ran- dom forests, with a simple and limited feature set, succeeds in modeling the complex deci- sions required to assess translation quality and achieves results that are on a par with the sec- ond best results of the shared task. 1 Introduction Confidence estimation is the task of predicting the quality of a system prediction without knowledge of the expected output. It is an important step in many Natural Language Processing applications (Gandrabur et al., 2006). In Machine Translation (MT), this task has recently gained interest (Blatz et al., 2004; Specia et al., 2010b; Soricut and Echi- habi, 2010; Bach et al., 2011). Indeed, professional translators are more and more requested to post-edit the outputs of a MT system rather than to produce a translation from scratch. Knowing in advance the segments they should focus on would be very help- ful (Specia et al., 2010a). Confidence estimation is also of great interest for developers of MT system, as it provides them with a way to analyze the systems output and to better understand the main causes of errors. Even if several studies have tackled the problem of confidence estimation in machine translation, un- til now, very few datasets were publicly available and comparing the proposed methods was difficult, if not impossible. To address this issue, WMT’12 orga- nizers proposed a shared task aiming at predict the ∗This work was conducted during an internship at LIMSI– CNRS quality of a translation and provided the associated datasets, baselines and metrics. This paper describes our work with the data of the WMT’12 Confidence Estimation shared task. Our contribution is twofold: i) we first present an analysis of the provided data that will stress the difficulty of the task and motivate the choice of our approach; ii) we show how using non-linear models, namely ran- dom forests, with a simple and limited features set succeed in modeling the complex decisions require to assess translation quality and achieve the second best results of the shared task. The rest of this paper is organized as follows: Sec- tion 2 summarizes our analysis of the data; in Sec- tion 3, we describe our learning method; our main results are finally reported in Section 4. 2 Data Analysis In this section, we quickly analyze the data dis- tributed in the context of the WMT’12 Confidence Estimation Shared Task in order to evaluate the …

Content truncated to 500 words