APE at Scale and Its Implications on MT Evaluation Biases

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 34–44 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 34 APE at Scale and its Implications on MT Evaluation Biases Markus Freitag, Isaac Caswell, Scott Roy Google Research {freitag,icaswell,hsr}@google.com Abstract In this work, we train an Automatic Post- Editing (APE) model and use it to reveal bi- ases in standard Machine Translation (MT) evaluation procedures. The goal of our APE model is to correct typical errors introduced by the translation process, and convert the “translationese” output into natural text. Our APE model is trained entirely on monolin- gual data that has been round-trip translated through English, to mimic errors that are similar to the ones introduced by NMT. We apply our model to the output of existing NMT systems, and demonstrate that, while the human-judged quality improves in all cases, BLEU scores drop with forward-translated test sets. We verify these results for the WMT18 English→German, WMT15 English→French, and WMT16 English→Romanian tasks. Fur- thermore, we selectively apply our APE model on the output of the top submissions of the most recent WMT evaluation campaigns. We see quality improvements on all tasks of up to 2.5 BLEU points. 1 Introduction Neural Machine Translation (NMT) (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) is currently the most popular approach in Machine Translation leading to state-of-the-art performance for many tasks. NMT relies mainly on parallel training data, which can be an expen- sive and scarce resource. There are several ap- proaches to leverage monolingual data for NMT: Language model fusion for both phrase-based (Brants et al., 2007) and neural MT (G¨ulc¸ehre et al., 2015, 2017), back-translation (Sennrich et al., 2016b), unsupervised NMT (Lample et al., 2018a; Artetxe et al., 2018a), dual learning (Cheng et al., 2016; He et al., 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017). In this paper, we present a different approach to leverage monolingual data, which can be used as a post-processor for any existing translation. The idea is to train an Automatic Post-Editing (APE) system that is only trained on a large amount of synthetic data, to fix typical errors introduced by the translation process. During training, our model uses a noisy version of each sentence as input and learns how to reconstruct the origi- nal sentence. In this work, we model the noise with round-trip translations (RTT) through En- glish, translating a sentence in the target language into English, then translating the result back into the original language. We train our APE model with a standard transformer model on the WMT18 English→German, WMT15 English→French and WMT16 English→Romanian monolingual News Crawl data and apply this model on the output of NMT models that are either trained on all avail- able bitext or trained on a combination of bitext and back-translated monolingual data. Further- more, we show that our APE model can be used as a post-processor for the best output of the recent WMT evaluation …

Content truncated to 500 words