Combining Quality Estimation and Automatic Post-editing to Enhance Machine Translation output

Combining Quality Estimation and Automatic Post-editing to Enhance Machine Translation Output Rajen Chatterjee chatterjee@fbk.eu Matteo Negri negri@fbk.eu Marco Turchi turchi@fbk.eu Fondazione Bruno Kessler, Trento, Italy Fr´ed´eric Blain f.blain@sheffield.ac.uk Lucia Specia l.specia@sheffield.ac.uk University of Sheffield, Sheffield, UK Abstract We investigate different strategies for combining quality estimation (QE) and automatic post- editing (APE) to improve the output of machine translation (MT) systems. The joint contribu- tion of the two technologies is analyzed in different settings, in which QE serves as either: i) an activator of APE corrections, or ii) a guidance to APE corrections, or iii) a selector of the final output to be returned to the user. In the first case (QE as activator), sentence-level predictions on the raw MT output quality are used to trigger its automatic correction when the estimated (TER) scores are below a certain threshold. In the second case (QE as guidance), word-level binary quality predictions (“good”/“bad”) are used to inform APE about problematic words in the MT output that should be corrected. In the last case (QE as selector), both sentence- and word-level quality predictions are used to identify the most accurate translation between the original MT output and its post-edited version. For the sake of comparison, the underlying APE technologies explored in our evaluation are both phrase-based and neural. Experiments are carried out on the English-German data used for the QE/APE shared tasks organized within the First Conference on Machine Translation (WMT 2016). Our evaluation shows positive but mixed results, with higher performance observed when word-level QE is used as a selec- tor for neural APE applied to the output of a phrase-based MT system. Overall, our findings motivate further investigation on QE technologies. By reducing the gap between the perfor- mance of current solutions and “oracle” results, QE could significantly add to competitive APE technologies. 1 Introduction In recent years, the steady progress of machine translation (MT) technology motivated research on a number of ancillary tasks. MT’s wide adoption, especially in the translation industry, has in fact raised new challenges, which are not only related to model training, optimization, adaptation and evaluation, but also to aspects that are external to the core translation approach. Among them, quality estimation (QE) and automatic post-editing (APE) deal respectively with the possibility of predicting the quality of MT output and correcting it via downstream post- processing. QE (Specia et al., 2010) is motivated by the need for estimating output quality at Proceedings of AMTA 2018, vol. 1: MT Research Track Boston, March 17 - 21, 2018 | Page 26 run-time, that is when reference-based evaluation is unfeasible (the typical scenario when MT is deployed in production). APE (Simard et al., 2007) is motivated by the need of improving MT systems’ output in black-box conditions, in which the translation models are not accessible for internal modification, retraining or adaptation (a typical situation for companies that rely on third-party MT systems). Both QE and APE have been successfully explored as standalone tasks in previous work, in particular within the well-established …

Content truncated to 500 words