On the Interaction of Regularization Factors in Low-resource Neural Machine Translation

On the Interaction of Regularization Factors in Low-resource Neural Machine Translation `Alex R. Atrio1,2 and Andrei Popescu-Belis1,2 1HEIG-VD / HES-SO 2EPFL Yverdon-les-Bains Lausanne Switzerland Switzerland {alejandro.ramirezatrio, andrei.popescu-belis}@heig-vd.ch Abstract We explore the roles and interactions of the hyper-parameters governing regulari- zation, and propose a range of values ap- plicable to low-resource neural machine translation. We demonstrate that default or recommended values for high-resource settings are not optimal for low-resource ones, and that more aggressive regulariza- tion is needed when resources are scarce, in proportion to their scarcity. We ex- plain our observations by the generaliza- tion abilities of sharp vs. flat basins in the loss landscape of a neural network. Re- sults for four regularization factors corrob- orate our claim: batch size, learning rate, dropout rate, and gradient clipping. More- over, we show that optimal results are ob- tained when using several of these fac- tors, and that our findings generalize across datasets of different sizes and languages. 1 Introduction The training of neural machine translation (NMT) models is governed by many hyper-parameters, which play a central role in the performances of the trained models, especially their generalization abilities. While most of the NMT frameworks rec- ommend default values for the hyper-parameters, when it comes to low-resource settings, fewer guidelines are available. This study systematically explores the roles and interactions of a subset of hyper-parameters in low-resource NMT settings, namely those acting © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. as regularization factors. Regularizers do not fall under a single theoretical definition: Goodfellow et al. (2016, page 224) view them as a collection of methods “intended to reduce generalization er- ror but not training error.” We present here a uni- fied perspective on several regularizers which act upon the estimation of the gradients during back- propagation. Using the distinction made by Keskar et al. (2016) between flat and sharp basins in the loss landscape, we argue that noisier estimates of the gradients can increase the likelihood of find- ing flatter minima, which have better generaliza- tion abilities. Specifically, we defend three claims: 1. NMT models benefit from more aggressive re- gularization when the amount of training data is small. We demonstrate this for four different reg- ularizers: batch size, learning rate, dropout, and gradient clipping. We compare the default regu- larization hyper-parameters of the OpenNMT-py framework for mid-to-high resources – compara- ble to those of the original Transformer (Vaswani et al., 2017) – with the ones we optimized for a low-resource setting (Sections 4-7). 2. The combination of different regularization sources is preferable over their individual use. When used together, an amount of regularization from each of the four factors under study outper- forms the use of any single one alone, and the best scores are robust with respect to the variation of each factor (Section 8). 3. Regularization factors optimized on one low- resource dataset are beneficial for low-resource datasets in other languages, and benefit from more aggressive …

Content truncated to 500 words