Dynamic Data Selection for Neural Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1400–1410 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Dynamic Data Selection for Neural Machine Translation Marlies van der Wees Informatics Institute University of Amsterdam Arianna Bisazza∗ LIACS Leiden University Christof Monz Informatics Institute University of Amsterdam Abstract Intelligent selection of training data has proven a successful technique to simul- taneously increase training efficiency and translation performance for phrase-based machine translation (PBMT). With the recent increase in popularity of neural machine translation (NMT), we explore in this paper to what extent and how NMT can also benefit from data selection. While state-of-the-art data selection (Ax- elrod et al., 2011) consistently performs well for PBMT, we show that gains are substantially lower for NMT. Next, we in- troduce dynamic data selection for NMT, a method in which we vary the selected sub- set of training data between different train- ing epochs. Our experiments show that the best results are achieved when applying a technique we call gradual fine-tuning, with improvements up to +2.6 BLEU over the original data selection approach and up to +3.1 BLEU over a general baseline. 1 Introduction Recent years have shown a rapid shift from phrase-based (PBMT) to neural machine transla- tion (NMT) (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014) as the most common machine translation paradigm. With large quan- tities of parallel data, NMT outperforms PBMT for an increasing number of language pairs (Bo- jar et al., 2016). Unfortunately, training an NMT model is often a time-consuming task, with train- ing times of several weeks not being unusual. Despite its training inefficiency, most work in NMT greedily uses all available training data for a given language pair. However, it is unlikely ∗Work done while at University of Amsterdam that all data is equally helpful to create the best- performing system. In PBMT, this issue has been addressed by applying intelligent data selection, and it has consistently been shown that using more data does not always improve translation quality (Moore and Lewis, 2010; Axelrod et al., 2011; Gasc´o et al., 2012). Instead, for a given translation task, the training bitext likely contains sentences that are irrelevant or even harmful, making it ben- eficial to keep only the most relevant subset of the data while discarding the rest, with the additional benefit of smaller models and faster training. Motivated by the success of data selection in PBMT, we investigate in this paper to what ex- tent and how NMT can benefit from data selec- tion as well. While data selection has been ap- plied to NMT to reduce the size of the data (Cho et al., 2014; Luong et al., 2015b), the effects on translation quality have not been investigated. In- tuitively, and confirmed by our exploratory exper- iments in Section 5.1, this is a challenging task; NMT systems are known to under-perform when trained on limited parallel data (Zoph et al., 2016; Fadaee et al., 2017), and …

Content truncated to 500 words