deepQuest: A Framework for Neural-based Quality Estimation

Proceedings of the 27th International Conference on Computational Linguistics, pages 3146–3157 Santa Fe, New Mexico, USA, August 20-26, 2018. 3146 deepQuest: A Framework for Neural-based Quality Estimation Julia Ive1 Fr´ed´eric Blain2 Lucia Specia2 King’s College London, IoPPN, London, SE5 8AF, UK1 Department of Computer Science, University of Sheffield Regent Court, 211 Portobello, Sheffield, S1 4DP, UK2 julia.ive@kcl.ac.uk, {f.blain, l.specia}@sheffield.ac.uk Abstract Predicting Machine Translation (MT) quality can help in many practical tasks such as MT post- editing. The performance of Quality Estimation (QE) methods has drastically improved recently with the introduction of neural approaches to the problem. However, thus far neural approaches have only been designed for word and sentence-level prediction. We present a neural framework that is able to accommodate neural QE approaches at these fine-grained levels and generalize them to the level of documents. We test the framework with two sentence-level neural QE ap- proaches: a state of the art approach that requires extensive pre-training, and a new light-weight approach that we propose, which employs basic encoders. Our approach is significantly faster and yields performance improvements for a range of document-level quality estimation tasks. To our knowledge, this is the first neural architecture for document-level QE. In addition, for the first time we apply QE models to the output of both statistical and neural MT systems for a series of European languages and highlight the new challenges resulting from the use of neural MT. 1 Introduction Quality Estimation (QE) (Blatz et al., 2004; Specia et al., 2009) aims at predicting the quality of machine translation (MT) without human intervention. Most recent work has focused on QE to predict sentence- level post-editing (PE) effort, i.e. the process of manually correcting MT output to achieve publishable quality (Bojar et al., 2014; Bojar et al., 2015; Bojar et al., 2016a; Bojar et al., 2017). In this case, QE indicates to what extent a MT sentence needs post-editing. Document-level QE, on the other hand, scores or ranks documents according to their quality for fully automated MT usage scenarios where no post-editing can be performed, e.g. MT for gisting of news articles online. Recently, neural methods have been successfully exploited to improve QE performance. These meth- ods mostly rely on either complex architectures, require extensive pre-training, or need some feature engineering (Patel and M, 2016; Kim et al., 2017a; Martins et al., 2017a; Jhaveri et al., 2018). In ad- dition, these methods have only been developed for word, phrase and sentence-level QE. These cannot be directly used for document-level QE since this level requires to take into account the content of the document in its entirety. State-of-the-art document-level QE solutions still rely on non-neural methods, and extensive feature engineering (Scarton et al., 2016). In this paper we propose a neural framework that is able to accommodate any QE approach at a fine- grained level (e.g. a sentence-level approach), and to generalize it to learn document-level QE models. We test the framework using a state of the art neural sentence-level QE approach (Kim et al., …

Content truncated to 500 words