Regularization techniques for fine-tuning in neural machine translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1489–1494 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Regularization techniques for fine-tuning in neural machine translation Antonio Valerio Miceli Barone Barry Haddow Ulrich Germann Rico Sennrich School of Informatics, The University of Edinburgh {amiceli, bhaddow, ugermann}@inf.ed.ac.uk rico.sennrich@ed.ac.uk Abstract We investigate techniques for super- vised domain adaptation for neural ma- chine translation where an existing model trained on a large out-of-domain dataset is adapted to a small in-domain dataset. In this scenario, overfitting is a major challenge. We investigate a number of techniques to reduce overfitting and im- prove transfer learning, including regular- ization techniques such as dropout and L2- regularization towards an out-of-domain prior. In addition, we introduce tuneout, a novel regularization technique inspired by dropout. We apply these techniques, alone and in combination, to neural ma- chine translation, obtaining improvements on IWSLT datasets for English→German and English→Russian. We also inves- tigate the amounts of in-domain train- ing data needed for domain adaptation in NMT, and find a logarithmic relationship between the amount of training data and gain in BLEU score. 1 Introduction Neural machine translation (Bahdanau et al., 2015; Sutskever et al., 2014) has established itself as the new state of the art at recent shared transla- tion tasks (Bojar et al., 2016; Cettolo et al., 2016). In order to achieve good generalization accuracy, neural machine translation, like most other large machine learning systems, requires large amounts of training examples sampled from a distribution as close as possible to the distribution of the inputs seen during execution. However, in many applica- tions, only a small amount of parallel text is avail- able for the specific application domain, and it is therefore desirable to leverage larger out-domain datasets. Owing to the incremental nature of stochastic gradient-based training algorithms, a simple yet effective approach to transfer learning for neural networks is fine-tuning (Hinton and Salakhutdi- nov, 2006; Mesnil et al., 2012; Yosinski et al., 2014): to continue training an existing model which was trained on out-of-domain data with in- domain training data. This strategy was also found to be very effective for neural machine transla- tion (Luong and Manning, 2015; Sennrich et al., 2016b). Since the amount of in-domain data is typically small, overfitting is a concern. A common solution is early stopping on a small held-out in-domain validation dataset, but this reduces the amount of in-domain data available for training. In this paper, we show that we can make fine- tuning strategies for neural machine translation more robust by using several regularization tech- niques. We consider fine-tuning with varying amounts of in-domain training data, showing that improvements are logarithmic in the amount of in- domain data. We investigate techniques where domain adap- tation starts from a pre-trained out-domain model, and only needs to process the in-domain cor- pus. Since we do not need to process the large out-domain corpus during adaptation, this is suitable for scenarios where adaptation must be performed quickly or where …

Content truncated to 500 words