Reproducible and Efficient Benchmarks for Hyperparameter Optimization of Neural Machine Translation Systems

Reproducible and Efficient Benchmarks for Hyperparameter Optimization of Neural Machine Translation Systems Xuan Zhang and Kevin Duh Johns Hopkins University xuanzhang@jhu.edu, kevinduh@cs.jhu.edu Abstract Hyperparameter selection is a crucial part of building neural machine translation (NMT) systems across both academia and industry. Fine-grained adjustments to a model’s archi- tecture or training recipe can mean the differ- ence between a positive and negative research result or between a state-of-the-art and under- performing system. While recent literature has proposed methods for automatic hyperparame- ter optimization (HPO), there has been limited work on applying these methods to neural machine translation (NMT), due in part to the high costs associated with experiments that train large numbers of model variants. To facilitate research in this space, we introduce a lookup-based approach that uses a library of pre-trained models for fast, low cost HPO experimentation. Our contributions include (1) the release of a large collection of trained NMT models covering a wide range of hyperparam- eters, (2) the proposal of targeted metrics for evaluating HPO methods on NMT, and (3) a reproducible benchmark of several HPO meth- ods against our model library, including novel graph-based and multiobjective methods. 1 Introduction Choosing effective hyperparameters is crucial for building strong neural machine translation (NMT) systems. Although some choices present obvious trade-offs (e.g., more and larger layers tend to increase quality at the cost of speed), others are more subtle (e.g., effects of batch size, learning rate, and normalization techniques on different layer types). Optimal versus suboptimal hyperparameters can lead to dramatic swings in system performance; consider the wide range of BLEU scores for variants of the same base system in Figure 1 (left). In practice, these hyperparame- ters are often tuned manually based on intuition and heuristics, a tedious and error-prone process that can lead to unreliable experimental results and underperforming shared task or production systems. The difficulty is compounded when system builders must jointly optimize multiple objectives, such as translation accuracy (BLEU) and decoding speed, which are largely uncorrelated as shown in Figure 1 (right). In the past decade, various hyperparameter optimization (HPO) methods have emerged in the machine learning literature under the labels of ‘‘AutoML’’ (Bergstra et al., 2011; Hutter et al., 2011; Bardenet et al., 2013; Snoek et al., 2015) and ‘‘neural architecture search’’ (Zoph and Le, 2016; Liu et al., 2018a,b; Cai et al., 2018; Real et al., 2019). However, it is unclear how they perform on NMT; we are not aware of any prior work with comprehensive evaluation. One challenge is that the state-of-the-art NMT models (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) require significant computational resources for training. Secondly, they usually have large hyperparameter search spaces. Thus, it is prohibitively expensive in practice to compare HPO methods on NMT tasks. In order to enable reproducible HPO research on NMT tasks, we adopt a benchmark procedure based on ‘‘table-lookup’’. This approach was recently introduced to neural architecture search by Ying et al. (2019), and to …

Content truncated to 500 words