Evolution Strategy Based Automatic Tuning of Neural Machine Translation Systems

Evolution Strategy Based Automatic Tuning of Neural Machine Translation Systems Hao Qin1, Takahiro Shinozaki1, Kevin Duh2 1Tokyo Institute of Technology, Japan 2Johns Hopkins University, USA qin.h.aa@m.titech.ac.jp, shinot@ict.e.titech.ac.jp, kevinduh@cs.jhu.edu Abstract Neural machine translation (NMT) systems have demon- strated promising results in recent years. However, non- trivial amounts of manual effort are required for tun- ing network architectures, training configurations, and pre- processing settings such as byte pair encoding (BPE). In this study, we propose an evolution strategy based automatic tun- ing method for NMT. In particular, we apply the covariance matrix adaptation-evolution strategy (CMA-ES), and investi- gate a Pareto-based multi-objective CMA-ES to optimize the translation performance and computational time jointly. Ex- perimental results show that the proposed method automati- cally finds NMT systems that outperform the initial manual setting. 1. Introduction Neural machine translation (NMT) is a new approach to translation and has shown promising results in recent years. Many active ongoing research are focused on developing new architectures and training methods. When developing a ma- chine translation system based on neural network structure, the major design question is how to set the meta-parameter values of the network structure and training configuration, so that the system performs well in terms of translation per- formance and computational cost. For network structure de- sign, important meta-parameters include what kind of net- work should be applied, the number of layers, the number of units per layer, and unit type. With the increase of layers, the problem becomes more complex. For training config- urations, important meta-parameters include learning algo- rithm, learning rate, dropout ratio and so on. All of these meta-parameters interact with each other and affect the per- formance of the whole system in a subtle way, thus they need to be tuned simultaneously. Usually, these meta-parameters are tuned by human ex- perts based on their experience. Such work requires much effort. In some ways such bottleneck may limit the wider adoption of NMT, or lead us into locally-optimal design de- cisions. Meanwhile, more powerful computing resource are available for academic and public use. Our motivation is to replace tedious manual tuning work with automatic compu- tation conducted by computers.1 As neural network training process is conducted off-line and a well-trained model can be used repeatedly, it is meaningful to allocate more computa- tional resources to meta-parameter tuning as it can alleviate manual work. Grid search is a simple method for meta-parameter op- timization. However, as the number of meta-parameters in- creases, it becomes less tractable. This is because the num- ber of lattice points increases in an exponential way with the increase of the number of meta-parameters. For example, if there are ten meta-parameters to be tuned and we only try five values for each parameter, it requires more than 750 billion (510) evaluations. In the case of NMT, training and evaluat- ing one instance requires significant computational resource and time. Thus grid search is not a feasible method even us- ing the fastest super computer. A black box meta-parameter optimization framework that …

Content truncated to 500 words