Reinforcement Learning based Curriculum Optimization for Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 2054–2061 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2054 Reinforcement Learning based Curriculum Optimization for Neural Machine Translation Gaurav Kumar Johns Hopkins University gkumar@cs.jhu.edu George Foster, Colin Cherry, Maxim Krikun Google AI {fosterg, colincherry, krikun} @google.com Abstract We consider the problem of making efficient use of heterogeneous training data in neu- ral machine translation (NMT). Specifically, given a training dataset with a sentence-level feature such as noise, we seek an optimal curriculum, or order for presenting examples to the system during training. Our curricu- lum framework allows examples to appear an arbitrary number of times, and thus gen- eralizes data weighting, filtering, and fine- tuning schemes. Rather than relying on prior knowledge to design a curriculum, we use re- inforcement learning to learn one automati- cally, jointly with the NMT system, in the course of a single training run. We show that this approach can beat uniform baselines on Paracrawl and WMT English-to-French datasets by +3.4 and +1.3 BLEU respectively. Additionally, we match the performance of strong filtering baselines and hand-designed, state-of-the-art curricula. 1 Introduction Machine Translation training data is typically het- erogeneous: it may vary in characteristics such as domain, translation quality, and degree of dif- ficulty. Many approaches have been proposed to cope with heterogeneity, such as filtering (Duh et al., 2013) or down-weighting (Wang et al., 2017) examples that are likely to be noisy or out of domain. A powerful technique is to control the curriculum—the order in which examples are pre- sented to the system—as is done in fine-tuning (Freitag and Al-Onaizan, 2016), where training occurs first on general data, and then on more valuable in-domain data. Curriculum based ap- proaches generalize data filtering and weighting1 by allowing examples to be visited multiple times 1Assuming integer weights. or not at all; and they additionally potentially en- able steering the training trajectory toward a better global optimum than might be attainable with a static attribute-weighting scheme. Devising a good curriculum is a challenging task that is typically carried out manually using prior knowledge of the data and its attributes. Although powerful heuristics like fine-tuning are helpful, setting hyper-parameters to specify a cur- riculum is usually a matter of extensive trial and error. Automating this process with meta-learning is thus an attractive proposition. However, it comes with many potential pitfalls such as failing to match a human-designed curriculum, or signif- icantly increasing training time. In this paper we present an initial study on meta-learning an NMT curriculum. Starting from scratch, we attempt to match the performance of a successful non-trivial reference curriculum pro- posed by Wang et al. (2018), in which train- ing gradually focuses on increasingly cleaner data, as measured by an external scoring func- tion. Inspired by Wu et al. (2018), we use a reinforcement-learning (RL) approach involving a learned agent whose task is to choose a corpus bin, representing a given noise level, at each NMT training step. A challenging aspect of this task …

Content truncated to 500 words