Pushing the Limits of Translation Quality Estimation

Pushing the Limits of Translation Quality Estimation Andr´e F. T. Martins Unbabel Instituto de Telecomunicac¸˜oes Lisbon, Portugal andre.martins@unbabel.com Marcin Junczys-Dowmunt Adam Mickiewicz University in Pozna´n Pozna´n, Poland junczys@amu.edu.pl Fabio N. Kepler Unbabel L2F/INESC-ID, Lisbon, Portugal University of Pampa, Alegrete, Brazil kepler@unbabel.com Ram´on Astudillo Unbabel L2F/INESC-ID Lisbon, Portugal ramon@unbabel.com Chris Hokamp Dublin City University Dublin, Ireland chokamp@computing.dcu.ie Roman Grundkiewicz Adam Mickiewicz University in Pozna´n Pozna´n, Poland romang@amu.edu.pl Abstract Translation quality estimation is a task of growing importance in NLP, due to its poten- tial to reduce post-editing human effort in dis- ruptive ways. However, this potential is cur- rently limited by the relatively low accuracy of existing systems. In this paper, we achieve remarkable improvements by exploiting syn- ergies between the related tasks of word-level quality estimation and automatic post-editing. First, we stack a new, carefully engineered, neural model into a rich feature-based word- level quality estimation system. Then, we use the output of an automatic post-editing sys- tem as an extra feature, obtaining striking re- sults on WMT16: a word-level F MULT 1 score of 57.47% (an absolute gain of +7.95% over the current state of the art), and a Pearson correla- tion score of 65.56% for sentence-level HTER prediction (an absolute gain of +13.36%). 1 Introduction The goal of quality estimation (QE) is to evaluate a translation system’s quality without access to ref- erence translations (Blatz et al., 2004; Specia et al., 2013). This has many potential usages: informing an end user about the reliability of translated con- tent; deciding if a translation is ready for publish- ing or if it requires human post-editing; highlighting the words that need to be changed. QE systems are particularly appealing for crowd-sourced and pro- fessional translation services, due to their potential to dramatically reduce post-editing times and to save labor costs (Specia, 2011). The increasing interest in this problem from an industrial angle comes as no surprise (Turchi et al., 2014; de Souza et al., 2015; Martins et al., 2016; Kozlova et al., 2016). In this paper, we tackle word-level QE, whose goal is to assign a label of OK or BAD to each word in the translation (Figure 1). Past approaches to this problem include linear classifiers with handcrafted features (Ueffing and Ney, 2007; Bic¸ici, 2013; Shah et al., 2013; Luong et al., 2014), often combined with feature selection (Avramidis, 2012; Beck et al., 2013), recurrent neural networks (de Souza et al., 2014; Kim and Lee, 2016), and systems that com- bine linear and neural models (Kreutzer et al., 2015; Martins et al., 2016). We start by proposing a “pure” QE system (§3) consisting of a new, carefully en- gineered neural model (NEURALQE), stacked into a linear feature-rich classifier (LINEARQE). Along the way, we provide a rigorous empirical analysis to better understand the contribution of the several groups of features and to justify the architecture of the neural system. A second contribution of this paper is bring- ing in the related task of automatic post-editing (APE; Simard et al. (2007)), which …

Content truncated to 500 words