Modeling Easiness for Training Transformers with Curriculum Learning

Proceedings of Recent Advances in Natural Language Processing, pages 937–948 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_101 937 Modeling Easiness for Training Transformers with Curriculum Learning Leonardo Ranaldi (∗,•), Giulia Pucci(∗), Fabio Massimo Zanzotto(∗) (•) Idiap Research Institute, Martigny, Switzerland (∗) Department of Enterprise Engineering, University of Rome Tor Vergata, Rome, Italy [first name].[last name]@uniroma2.it Abstract Directly learning from complex examples is generally problematic for humans and ma- chines. Indeed, a better strategy is expos- ing learners to examples in a reasonable, pedagogically-motivated order. Curriculum Learning (CL) has been proposed to import this strategy when training machine learning models. In this paper, building on Curriculum Learning, we propose a novel, linguistically motivated measure to determine example com- plexity for organizing examples during learn- ing. Our complexity measure - LRC- is based on length, rarity, and comprehensibility. Our resulting learning model is CL-LRC, that is, CL with LRC. Experiments on downstream tasks show that CL-LRC outperforms existing CL and non-CL methods for training BERT and RoBERTa from scratch. Furthermore, we an- alyzed different measures, including perplex- ity, loss, and learning curve of different models pre-trained from scratch, showing that CL-LRC performs better than the state-of-the-art. 1 Introduction Pre-trained Transformers are sweeping away all other methods of natural language understanding. These models outperform all previous methods and sometimes even humans in many NLP tasks (Wang et al., 2018, 2020; Kalyan et al., 2021; Guo et al., 2022). Pre-training on unlabeled large-scale cor- pora seems to be the way that increases perfor- mance (Ranaldi et al., 2022). For example, BERT is pre-trained on an English corpus of 3.300 mil- lion words consisting of books (Zhu et al., 2015) and Wikipedia. However, training these models with large corpora is quite expensive in terms of computation time and memory. The problem of optimizing the computational resources that Transformers need is tackled in three main ways: by re-modeling pre-training tasks (Yang et al., 2019a; Clark et al., 2020), by studying techniques to produce lighter architectures (Sanh et al., 2019; Liu et al., 2019), and by working with data (Moore and Lewis, 2010; Gururangan et al., 2020; Chang et al., 2021). Architecture-level and model-level optimization techniques have been extensively studied in the context of pre-training methods for NLP. Data-level approaches have yet to be explored. To this end, we adopted a data-level strategy called Curriculum Learning (CL), which stems from the complexity of training samples so that the model can achieve better performances. Starting from the idea for which humans and animals acquire first elemental concepts and then, gradually, more complex ones, Bengio et al. (2009) proposed CL and demonstrated its benefits in shape recognition. This approach presents training data in order of difficulty, starting with easy examples and increasing the degree in parallel with learning. The application of CL in Pre-trained Language Models (PLMs) has limitations. One of the most critical challenges is to find a criterion for mea- suring the difficulty of training samples. In super- vised tasks, sorting training batches by length and repetitiveness of certain …

Content truncated to 500 words