Assessing Quality Estimation Models for Sentence-Level Prediction

Proceedings of the 27th International Conference on Computational Linguistics, pages 1521–1533 Santa Fe, New Mexico, USA, August 20-26, 2018. 1521 Assessing Quality Estimation Models for Sentence-Level Prediction Hoang Cuong and Jia Xu Cuny University of New York Abstract This paper provides an evaluation of a wide range of advanced sentence-level Quality Estimation models, including Support Vector Regression, Ride Regression, Neural Networks, Gaussian Pro- cesses, Bayesian Neural Networks, Deep Kernel Learning and Deep Gaussian Processes. Beside the accurateness, our main concerns are also the robustness of Quality Estimation models. Our work raises the difficulty in building strong models. Specifically, we show that Quality Estima- tion models often behave differently in Quality Estimation feature space, depending on whether the scale of feature space is small, medium or large. We also show that Quality Estimation mod- els often behave differently in evaluation settings, depending on whether test data come from the same domain as the training data or not. Our work suggests several strong candidates to use in different circumstances. 1 Introduction Quality Estimation (QE) (Blatz et al., 2004; Specia et al., 2009) is an important topic in Natural Language Processing (NLP). QE aims to predict the quality of Machine Translation (MT) outputs without human references. QE has a lot of potential. Such a case is automatically optimizing MT systems without having reference of translation outputs. Another example is MT for gisting by users of online translation systems. QE can also reduce post-editing human effort in disruptive ways. As most QE research has conducted at sentence level, we focus on the sentence level in this study. Prior work developed strong QE predictors by creating powerful QE features. Hand-craft and syn- tactic feature templates have been proposed for a while (e.g. see Specia et al. (2009) and Martins et al. (2017)). The representation of words, phrases and sentences can also be learned automatically using Neural Networks, serving as powerful QE features (Kreutzer et al., 2015; Shah et al., 2015b; Kim and Lee, 2016; Kim et al., 2017; Chen et al., 2017; Bic¸ici, 2017; Martins et al., 2017). Meanwhile, using a relevant QE model is also very important in QE. Various QE models have been applied in different QE settings, such as Support Vector Regression (Cortes and Vapnik, 1995; Specia et al., 2009), Ride Regression (Hoerl and Kennard, 2000; Wisniewski et al., 2013), Neural Networks (NNs) (Avramidis, 2017; Paetzold and Specia, 2016; Paetzold and Specia, 2017) and Gaussian Processes (GPs) (Rasmussen and Williams, 2005; Cohn and Specia, 2013). Given that there are many powerful regression models, it is tempting to have an understanding of which sentence-level QE models we should use. Prior work has been attempted to provide an answer (e.g. see Cohn and Specia (2013), Soricut et al. (2012), Wisniewski et al. (2013), and Beck et al. (2016)). However, the assessment of QE models is often limited in the number of QE features. For instance, Cohn and Specia (2013) conduct experiments with only 17 QE features. Also, the assessment is in only the Standard …

Content truncated to 500 words