Pre-training a BERT with Curriculum Learning by Increasing Block-Size of Input Text

Proceedings of Recent Advances in Natural Language Processing, pages 989–996 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_112 989 Pre-training a BERT with Curriculum Learning by Increasing Block-Size of Input Text Koichi Nagatsuka Soka University Tokyo, Japan e20d5301@soka-u.jp Clifford Broni-Bediako Soka University Tokyo, Japan e18d5252@soka-u.jp Masayasu Atsumi Soka University Tokyo, Japan matsumi@soka.ac.jp Abstract Recently, pre-trained language representation models such as BERT and RoBERTa have achieved significant results in a wide range of natural language processing (NLP) tasks, how- ever, it requires extremely high computational cost. Curriculum learning (CL) is one of the potential solutions to alleviate this problem. CL is a training strategy where training sam- ples are given to models in a meaningful or- der instead of random sampling. In this work, we propose a new CL method which gradu- ally increases the block-size of input text for training the self-attention mechanism of BERT and its variants using the maximum available batch-size. Experiments in low-resource set- tings show that our approach outperforms the baseline in terms of convergence speed and fi- nal performance on down-stream tasks. 1 Introduction Recent years have seen a series of breakthroughs in pre-trained language representation models. The development of pre-training methods like BERT (Devlin et al., 2019) and its variants (Liu et al., 2019) have led to large improvements in many down-stream tasks such as paraphrase identifica- tion, sentence textual similarity, sentiment analysis, and natural language inference. One of the ad- vantages in training these models is that they can leverage the unlabeled large-scale corpora which are more available compared to the labeled ones. However, training these models with large-scale corpora is pretty expensive in terms of computa- tional time and memory footprint. In the litera- ture, there are three main approaches that have been adopted to address this problem. These are architecture-based approach (Sanh et al., 2019; Voita et al., 2019; Sukhbaatar et al., 2019; de Wyn- ter and Perry, 2020; Lan et al., 2020), task-based approach (Yang et al., 2019; Clark et al., 2020) and dataset-based approach (Elman, 1993; Bengio et al., 2009; Moore and Lewis, 2010; Gururangan et al., 2020). While the architecture-based and task- based methods have been extensively studied in the context of pre-training methods for natural lan- guage processing (NLP), dataset-based approach is relatively unexplored. To this end, we adopt a dataset-based method called Curriculum Learning (CL) which controls the order of training samples so that the model might converge faster with better performance. The idea of CL-like approach was originally pro- posed by Elman (1993). The idea is based on the actual learning mechanism of humans and animals, where basic concepts are acquired first, then more complex ones are gradually learned. Bengio et al. (2009) formalized this concept as CL to train neu- ral networks. Through experimental analysis, Ben- gio et al. (2009) showed the benefit of CL on con- vergence speed and performance in shape recog- nition and language modeling tasks. One of the most significant challenges when adapting CL to a new task is to figure out a criterion …

Content truncated to 500 words