Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1832–1838 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1832 Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning Seonghyeon Ye, Jiseon Kim, Alice Oh School of Computing, KAIST {vano1205, jiseon_kim}@kaist.ac.kr alice.oh@kaist.edu Abstract We introduce EfficientCL, a memory-efficient continual pretraining method that applies con- trastive learning with novel data augmentation and curriculum learning. For data augmen- tation, we stack two types of operation se- quentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curricu- lum learning by incrementing the augmenta- tion degree for each difficulty step. After data augmentation, we apply contrastive learning on projected embeddings of original and aug- mented examples. When fine-tuned on GLUE benchmark, our model outperforms baseline models, especially for sentence-level tasks. Additionally, this improvement is achieved with only 70% of computational memory com- pared to the baseline model. 1 1 Introduction Many state-of-the-art language models involve the paradigm of unsupervised pretraining followed by fine-tuning on downstream tasks (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020). How- ever, pretraining a language model from scratch with a huge corpus has high computational costs. One way to cut down the cost is to use continual training of a pretrained model which could improve the language model with less computation (Giorgi et al., 2021). Contrastive learning is effective for self- supervised learning for image classification (Chen et al., 2020b,c), and it works by allowing the model to put similar examples close and different exam- ples far from one another. Often in contrastive learning, data augmentation is used to make the positive pairs. Recent papers describe how to apply contrastive learning to the language domain (Meng et al., 2021; Gunel et al., 2020; Qu et al., 2020; Wu et al., 2020), and even a combination of contrastive 1Our code is publicly available at https://github. com/vano1205/EfficientCL learning with continual pretraining (Giorgi et al., 2021). However, because of the sequential nature of language, it is difficult to apply data augmenta- tion methods used in images directly to language modeling. Additionally, curriculum learning is a powerful training technique for deep networks (Hacohen and Weinshall, 2019; Cai et al., 2018). By training easy to hard examples in order, it facilitates faster convergence, leading to better performance. Other studies show that it is also effective for language modeling (Xu et al., 2020; Wei et al., 2021; Press et al., 2021; Li et al., 2021). We propose an efficient yet powerful continual pretraining method using contrastive learning and curriculum learning. The contribution of this paper is as follows: • We suggest a novel data augmentation method for contrastive learning: first cutoff, then PCA jittering. This leads to robustness to sentence- level noise, resulting in a better sentence-level representation. • We apply curriculum learning by increasing the noise degree of augmentation for each level of difficulty. This leads to faster con- vergence at the pretraining stage. • In addition to outperforming baseline models on the GLUE benchmark, …

Content truncated to 500 words