Massive Exploration of Neural Machine Translation Architectures

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1442–1451 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Massive Exploration of Neural Machine Translation Architectures Denny Britz∗†, Anna Goldie∗, Minh-Thang Luong, Quoc V. Le {dennybritz,agoldie,thangluong,qvl}@google.com Google Brain Abstract Neural Machine Translation (NMT) has shown remarkable progress over the past few years, with production systems now being deployed to end-users. As the field is moving rapidly, it has become un- clear which elements of NMT architec- tures have a significant impact on trans- lation quality. In this work, we present a large-scale analysis of the sensitivity of NMT architectures to common hyper- parameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on a WMT English to German translation task. Our experiments provide practical insights into the relative importance of factors such as embedding size, network depth, RNN cell type, resid- ual connections, attention mechanism, and decoding heuristics. As part of this con- tribution, we also release an open-source NMT framework in TensorFlow to make it easy for others to reproduce our results and perform their own experiments. 1 Introduction Neural Machine Translation (NMT) (Kalchbren- ner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014) is an end-to-end approach to machine translation. NMT has shown impressive results (Jean et al., 2015; Luong et al., 2015b; Sennrich et al., 2016a; Wu et al., 2016) surpass- ing those of phrase-based systems while address- ing shortcomings, such as the need for hand- engineered features. The most popular approaches ∗Both authors contributed equally to this work. †Work done as a member of the Google Brain Residency program (g.co/brainresidency). to NMT are based on sequence-to-sequence mod- els, an encoder-decoder architecture consisting of two recurrent neural networks (RNNs) and an attention mechanism that aligns target with source tokens (Bahdanau et al., 2015; Luong et al., 2015a). One drawback of current NMT architectures is the huge amount of compute required to train them. Training on real-world datasets of sev- eral million examples typically requires dozens of GPUs and convergence time is on the order of days to weeks (Wu et al., 2016). While sweeping across large hyperparameter spaces is common in Com- puter Vision (Huang et al., 2016b), such explo- ration would be prohibitively expensive for NMT models, limiting researchers to well-established architecture and hyperparameter choices. Further- more, there have been no large-scale studies of how these hyperparameters affect the performance of NMT systems. As a result, it remains unclear why these models perform as well as they do or how we might improve them. In this work, we present an extensive analysis of architectural hyperparameters for NMT systems. Using a total of more than 250,000 GPU hours, we explore common variations of NMT architec- tures and provide insights into which architectural choices matter most. We report BLEU scores, per- plexities, model sizes, and convergence time for all experiments, including variance numbers cal- culated across several runs of each experiment. In …

Content truncated to 500 words