Can Automatic Post-Editing Improve NMT?

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2736–2746, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2736 Can Automatic Post-Editing Improve NMT? Shamil Chollampatt Rakuten, Inc. shamil.chollampatt@rakuten.com Raymond Hendy Susanto Rakuten, Inc. raymhs91@gmail.com Liling Tan Rakuten, Inc. liling.tan@rakuten.com Ewa Szymanska Rakuten, Inc. ewa.szymanska@rakuten.com Abstract Automatic post-editing (APE) aims to improve machine translations, thereby reducing human post-editing effort. APE has had notable suc- cess when used with statistical machine trans- lation (SMT) systems but has not been as suc- cessful over neural machine translation (NMT) systems. This has raised questions on the relevance of APE task in the current sce- nario. However, the training of APE models has been heavily reliant on large-scale artifi- cial corpora combined with only limited hu- man post-edited data. We hypothesize that APE models have been underperforming in improving NMT translations due to the lack of adequate supervision. To ascertain our hy- pothesis, we compile a larger corpus of hu- man post-edits of English to German NMT. We empirically show that a state-of-art neural APE model trained on this corpus can signifi- cantly improve a strong in-domain NMT sys- tem, challenging the current understanding in the field. We further investigate the effects of varying training data sizes, using artificial training data, and domain specificity for the APE task. We release this new corpus un- der CC BY-NC-SA 4.0 license at https:// github.com/shamilcm/pedra. 1 Introduction Automatic Post-Editing (APE) aims to reduce manual post-editing effort by automatically fixing errors in the machine-translated output. Knight and Chander (1994) first proposed APE to cope with systematic errors in selecting appropriate ar- ticles for Japanese to English translation. Earlier application of statistical phrase-based models for APE treated it as a monolingual re-writing task without considering the source sentence (Simard et al., 2007; B´echara et al., 2011). Modern APE models take the source text and machine- translated text as input and output the post-edited text in the target language (see Figure 1). Source text (English): Will he send the gifts to the house? Machine translated text (German): Die Geschenke in mein Haus schicken? (The gifts) (to my) (house) (send) Post-edited text (German): Wird er die Geschenke ins Haus schicken? (Will he) (the gifts) (to the) (house) (send) Figure 1: An example of post-editing given the source text in English and the translated text in German. APE models are usually trained and evaluated in a black-box scenario where the underlying MT model and the decoding process are inaccessible making it difficult to improve the MT system di- rectly. APE can be effective in this case to improve the MT output or to adapt its style or domain. Recent advancement of APE has shown remark- able success on statistical machine translation (SMT) outputs (Junczys-Dowmunt and Grund- kiewicz, 2018; Correia and Martins, 2019) even when trained with limited number of post-edited training instances (generally “triplets” consisting of source, translated, and post-edited segments), with or without additional large-scale artificial data (Junczys-Dowmunt and Grundkiewicz, 2016; Negri et al., 2018). Substantial improvements …

Content truncated to 500 words