Scaling Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 1–9 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64001 Scaling Neural Machine Translation Myle Ott△ Sergey Edunov△ David Grangier▽∗ Michael Auli△ △Facebook AI Research, Menlo Park & New York. ▽Google Brain, Mountain View. Abstract Sequence to sequence learning models still re- quire several days to reach state of the art per- formance on large benchmark datasets using a single machine. This paper shows that re- duced precision and large batch training can speedup training by nearly 5x on a single 8- GPU machine with careful tuning and im- plementation.1 On WMT’14 English-German translation, we match the accuracy of Vaswani et al. (2017) in under 5 hours when training on 8 GPUs and we obtain a new state of the art of 29.3 BLEU after training for 85 minutes on 128 GPUs. We further improve these results to 29.8 BLEU by training on the much larger Paracrawl dataset. On the WMT’14 English- French task, we obtain a state-of-the-art BLEU of 43.2 in 8.5 hours on 128 GPUs. 1 Introduction Neural Machine Translation (NMT) has seen im- pressive progress in the recent years with the intro- duction of ever more efficient architectures (Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Similar sequence-to-sequence mod- els are also applied to other natural language processing tasks, such as abstractive summariza- tion (See et al., 2017; Paulus et al., 2018) and dialog (Sordoni et al., 2015; Serban et al., 2017; Dusek and Jurc´ıcek, 2016). Currently, training state-of-the-art models on large datasets is computationally intensive and can require several days on a machine with 8 high- end graphics processing units (GPUs). Scaling training to multiple machines enables faster exper- imental turn-around but also introduces new chal- lenges: How do we maintain efficiency in a dis- tributed setup when some batches process faster *Work done while at Facebook AI Research. 1Our implementation is available at: https://www.github.com/pytorch/fairseq than others (i.e., in the presence of stragglers)? How do larger batch sizes affect optimization and generalization performance? While stragglers pri- marily affect multi-machine training, questions about the effectiveness of large batch training are relevant even for users of commodity hardware on a single machine, especially as such hardware continues to improve, enabling bigger models and batch sizes. In this paper, we first explore approaches to im- prove training efficiency on a single machine. By training with reduced floating point precision we decrease training time by 65% with no effect on accuracy. Next, we assess the effect of dramati- cally increasing the batch size from 25k to over 400k tokens, a necessary condition for large scale parallelization with synchronous training. We im- plement this on a single machine by accumulating gradients from several batches before each update. We find that by training with large batches and by increasing the learning rate we can further reduce training time by 40% on a single machine. Fi- nally, we parallelize …

Content truncated to 500 words