On the differences between human translations

Efficiently Reusing Old Models Across Languages via Transfer Learning Tom Kocmi Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostranské námˇestí 25, 118 00 Prague, Czech Republic {kocmi,bojar}@ufal.mff.cuni.cz Abstract Recent progress in neural machine transla- tion is directed towards larger neural net- works trained on an increasing amount of hardware resources. As a result, NMT mod- els are costly to train, both financially, due to the electricity and hardware cost, and en- vironmentally, due to the carbon footprint. It is especially true in transfer learning for its additional cost of training the “parent” model before transferring knowledge and training the desired “child” model. In this paper, we propose a simple method of re- using an already trained model for different language pairs where there is no need for modifications in model architecture. Our approach does not need a separate parent model for each investigated language pair, as it is typical in NMT transfer learning. To show the applicability of our method, we recycle a Transformer model trained by dif- ferent researchers and use it to seed models for different language pairs. We achieve better translation quality and shorter con- vergence times than when training from ran- dom initialization. 1 Introduction Neural machine translation (NMT), the current prevalent approach to automatic translation, is known to require large amounts of parallel training sentences and an extensive amount of training time on dedicated hardware. The total training time sig- nificantly increases, especially when training strong c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. baselines, searching for best hyperparameters or training multiple models for various language pairs. Schwartz et al. (2019) analyzed 60 papers from top AI conferences and found out that 80% of them target accuracy over efficiency, and only a small portion of papers argue for a new efficiency result. They also noted that the increasing financial cost of the computations could make it difficult for re- searchers to engage in deep learning research or limit training strong baselines. Furthermore, in- creased computational requirements have also an environmental cost. Strubell et al. (2019) estimated that training a single Transformer “big” model pro- duces 87 kg of CO2 and that the massive Trans- former architecture parameter search produced 298 tonnes of CO2.1 However, a lot of research has been already in- vested into cutting down the long training time by the design of NMT model architectures, promot- ing self-attentive (Vaswani et al., 2017) or convo- lutional (Gehring et al., 2017) over recurrent ones (Bahdanau et al., 2014) or the implementation of heavily optimized toolkits (Junczys-Dowmunt et al., 2018). In this paper, we propose a novel view on re- using already trained “parent” models without the need to prepare a parent model in advance or mod- ify its training hyper-parameters. Furthermore, we propose a second method based on a vocabulary transformation technique that makes even larger improvements, especially for languages using an alphabet different from …

Content truncated to 500 words