Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2595 - 2611 December 7-11, 2022 ©2022 Association for Computational Linguistics Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU Fenia Christopoulou, Gerasimos Lampouras, Ignacio Iacobacci Huawei Noah’s Ark Lab, London, UK {efstathia.christopoulou, gerasimos.lampouras, ignacio.iacobacci}@huawei.com Abstract Curriculum Learning (CL) is a technique of training models via ranking examples in a typi- cally increasing difficulty trend with the aim of accelerating convergence and improving gen- eralisability. Current approaches for Natural Language Understanding (NLU) tasks use CL to improve in-distribution data performance of- ten via heuristic-oriented or task-agnostic dif- ficulties. In this work, instead, we employ CL for NLU by taking advantage of training dy- namics as difficulty metrics, i.e. statistics that measure the behavior of the model at hand on specific task-data instances during training and propose modifications of existing CL sched- ulers based on these statistics. Differently from existing works, we focus on evaluating mod- els on in-distribution (ID), out-of-distribution (OOD) as well as zero-shot (ZS) cross-lingual transfer datasets. We show across several NLU tasks that CL with training dynamics can re- sult in better performance mostly on zero-shot cross-lingual transfer and OOD settings with improvements up by 8.5% in certain cases. Overall, experiments indicate that training dy- namics can lead to better performing models with smoother training compared to other dif- ficulty metrics while being 20% faster on av- erage. In addition, through analysis we shed light on the correlations of task-specific versus task-agnostic metrics1. 1 Introduction Transformer-based language models (Vaswani et al., 2017; Devlin et al., 2019, LMs) have re- cently achieved great success in a variety of NLP tasks (Wang et al., 2018, 2019a). However, gen- eralisation to out-of-distribution (OOD) data and zero-shot cross-lingual transfer still remain a chal- lenge (Linzen, 2020; Hu et al., 2020). Among exist- ing techniques, improving OOD performance has 1Code is available at https://github.com/ huawei-noah/noah-research/tree/master/ NLP/TD4CL been addressed by training with adversarial data (Yi et al., 2021), while better transfer across languages has been achieved by selecting appropriate lan- guages to transfer from (Lin et al., 2019; Turc et al., 2021), employing meta-learning (Nooralahzadeh et al., 2020) or data alignment (Fang et al., 2020). Contrastive to such approaches that take advan- tage of additional training data is Curriculum Learn- ing (Bengio et al., 2009, CL), a technique that aims to train models using a specific ordering of the original training examples. This ordering typically follows an increasing difficulty trend where easy examples are fed to the model first, moving to- wards harder instances. The intuition behind CL stems from human learning, as humans focus on simpler concepts before learning more complex ones, a procedure that is called shaping (Krueger and Dayan, 2009). Although curricula have been primarily used for Computer Vision (Hacohen and Weinshall, 2019; Wu et al., 2021) and Machine Translation (Zhang et al., 2019a; Platanios et al., 2019), there are only a handful of approaches that incorporate CL into Natural Language Understand- ing tasks …

Content truncated to 500 words