Tricks for Training Sparse Translation Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3340 - 3345 July 10-15, 2022 ©2022 Association for Computational Linguistics Tricks for Training Sparse Translation Models Dheeru Dua♥ Shruti Bhosale♠ Vedanuj Goswami♠ James Cross♠ Mike Lewis♠ Angela Fan♠ ♥University of California, Irvine, USA ♠Facebook AI ddua@uci.edu Abstract Multi-task learning with an unbalanced data distribution skews learning towards high re- source tasks, especially when model capac- ity is fixed and fully shared across all tasks. Sparse scaling architectures, such as BASE- Layers, provide flexible mechanisms for tasks to have a variable number of parameters, which can be useful to counterbalance skewed data distributions. However, we find that that BASELayers sparse model for multilingual machine translation can perform poorly out of the box, and propose two straightforward tech- niques to mitigate this — a temperature heat- ing mechanism and dense pre-training. Over- all, these methods improve performance on two multilingual translation benchmarks com- pared to standard BASELayers and dense scal- ing baselines, and in combination, more than 2x model convergence speed. 1 Introduction Training a universal model capable of handling many different tasks is a longstanding ambition in natural language processing (Collobert and We- ston, 2008; Ruder, 2017; McCann et al., 2018), with recent progress driven by training transformer models on a wide range of tasks (Xue et al., 2021; Khashabi et al., 2020; Lu et al., 2020). A central challenge in multi-task learning is accounting for the dramatically varying amounts of training data available for different tasks, which can lead to over- fitting on low-resource tasks whilst simultaneously underfitting on tasks with abundant training data. In this work, we study multilingual machine translation as a multi-task learning problem (Dong et al., 2015; Firat et al., 2016), where a single model is trained to translate between many language pairs (Fan et al., 2021). Multilingual learning has the potential of crosslingual transfer, allowing low- resource languages to benefit from high-resource data (Conneau et al., 2020). However, in practice, this positive transfer is often mitigated by interfer- ence between languages (Arivazhagan et al., 2019; Tan et al., 2019; Zhang et al., 2020). This is because all languages, irrespective of the amount of data, are trained with a fixed model capacity (Lepikhin et al., 2020), leading to insufficient specialized ca- pacity. Recent efforts have focused on sparse ar- chitectures (Lewis et al., 2021) to train high ca- pacity models, but these overfit to low-resource languages and have worse performance than dense architectures (Fan et al., 2021; Tran et al., 2021). We analyze the learning patterns of experts through- out training and identify a fundamental problem: experts specialize early on and rarely change spe- cialization. We propose two straightforward techniques to improve BASELayers-based sparse architec- tures (Lewis et al., 2021) for multitask learning: first, we slowly ramp the number of instances from low-resource tasks over epochs rather than having a fixed sampling ratio (Arivazhagan et al., 2019). This promotes cross-lingual transfer and reduces over-fitting as …

Content truncated to 500 words