Generalizing Back-Translation in Neural Machine Translation

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 45–52 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 45 Generalizing Back-Translation in Neural Machine Translation Miguel Grac¸a1† Yunsu Kim1 Julian Schamper1† Shahram Khadivi2 Hermann Ney1 1Human Language Technology and Pattern Recognition Group RWTH Aachen University, Aachen, Germany {surname}@i6.informatik.rwth-aachen.de 2eBay, Inc., Aachen, Germany {skhadivi}@ebay.com Abstract Back-translation — data augmentation by translating target monolingual data — is a crucial component in modern neural machine translation (NMT). In this work, we refor- mulate back-translation in the scope of cross- entropy optimization of an NMT model, clari- fying its underlying mathematical assumptions and approximations beyond its heuristic usage. Our formulation covers broader synthetic data generation schemes, including sampling from a target-to-source NMT model. With this for- mulation, we point out fundamental problems of the sampling-based approaches and propose to remedy them by (i) disabling label smooth- ing for the target-to-source model and (ii) sam- pling from a restricted search space. Our state- ments are investigated on the WMT 2018 Ger- man ↔ English news translation task. 1 Introduction Neural machine translation (NMT) (Bahdanau et al., 2014; Vaswani et al., 2017) systems make use of back-translation (Sennrich et al., 2016a) to leverage monolingual data during the train- ing. Here an inverse, target-to-source, transla- tion model generates synthetic source sentences, by translating a target monolingual corpus, which are then jointly used as bilingual data. Sampling-based synthetic data generation schemes were recently shown to outperform beam search (Edunov et al., 2018; Imamura et al., 2018). However, the generated corpora are reported to stray away from the distribution of natural data (Edunov et al., 2018). In this work, we focus on investigating why sampling creates better training data by re-writing the loss criterion of an NMT model to include a model-based data generator. † Now at DeepL GmbH. By doing so, we obtain a deeper understanding of synthetic data generation methods, identifying their desirable properties and clarifying the practical approximations. In addition, current state-of-the-art NMT mod- els suffer from probability smearing issues (Ott et al., 2018) and are trained using label smoothing (Pereyra et al., 2017). These result in low-quality sampled sentences, which influence the synthetic corpora. We investigate considering only high- quality hypotheses by restricting the search space of the model via (i) ignoring words under a prob- ability threshold during sampling and (ii) N-best list sampling. We validate our claims in experiments on a con- trolled scenario derived from the WMT 2018 Ger- man ↔ English translation task, which allows us to directly compare the properties of synthetic and natural corpora. Further, we present the proposed sampling techniques on the original WMT Ger- man ↔ English task. The experiments show that our restricted sampling techniques work compa- rable or superior to other generation methods by imitating human-generated data better. In terms of translation quality, these do not result in consistent improvements over the typical beam search strat- egy. 2 Related Work Sennrich et al. (2016a) introduce the back- translation …

Content truncated to 500 words