Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7417 - 7427 December 7-11, 2022 ©2022 Association for Computational Linguistics Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking Mingyu Lee1∗, Jun-Hyung Park2∗, Junho Kim1, Kang-Min Kim3, SangKeun Lee1,2 1Department of Artificial Intelligence 2Department of Computer Science and Engineering Korea University, Seoul, Republic of Korea 3Department of Data Science, The Catholic University of Korea, Bucheon, Republic of Korea {decon9201, irish07, monocrat}@korea.ac.kr kangmin89@catholic.ac.kr, yalphy@korea.ac.kr Abstract Masked language modeling (MLM) has been widely used for pre-training effective bidirec- tional representations, but incurs substantial training costs. In this paper, we propose a novel concept-based curriculum masking (CCM) method to efficiently pre-train a lan- guage model. CCM has two key differences from existing curriculum learning approaches to effectively reflect the nature of MLM. First, we introduce a carefully-designed linguistic dif- ficulty criterion that evaluates the MLM diffi- culty of each token. Second, we construct a curriculum that gradually masks words related to the previously masked words by retrieving a knowledge graph. Experimental results show that CCM significantly improves pre-training efficiency. Specifically, the model trained with CCM shows comparative performance with the original BERT on the General Lan- guage Understanding Evaluation benchmark at half of the training cost. Code is available at https://github.com/KoreaMGLEE/Concept- based-curriculum-masking. 1 Introduction Self-supervised pre-training has achieved consider- able performance improvements in various natural language processing (NLP) tasks (Devlin et al., 2019; Yang et al., 2019; Zhang et al., 2019; Lewis et al., 2020; Clark et al., 2020). Masked language modeling (MLM) (Devlin et al., 2019), where the model predicts original tokens of a masked subset of the text using the unmasked subset as clues, has contributed significantly to these improvements. However, MLM typically requires a high number of compute operations, resulting in unrealistically large training costs (Clark et al., 2020). Therefore, the consideration on the pre-training costs has be- come an increasingly important issue (Jiang et al., 2020; Narayanan et al., 2021). ∗These authors contributed equally to this work. Methods CoLA MRPC SST Avg. No curriculum 44.9 85.4 89.6 73.3 Rarity 45.5 85.1 89.4 73.4 Length 41.6 86.3 89.4 72.4 Masking ratio 43.2 87.0 89.7 73.3 CCM (ours) 48.0 86.7 90.9 75.2 Table 1: Comparison of different CL methods with BERTMedium. Rarity denotes a curriculum that first learns sentences composed with frequent words. Length denotes a curriculum that incrementally increases the sequence length from 64 to 512. Masking ratio denotes a curriculum that increases the masking ratio linearly from 0.1 to 0.15. Recent NLP studies have shown that curricu- lum learning (CL), presenting examples in an easy- to-difficult order rather than presenting them ran- domly, can accelerate the model convergence and improve the generalization performance (Zhang et al., 2018; Tay et al., 2019; Zhan et al., 2021). There mainly exist two criteria for assessing the dif- ficulty of examples, 1) model-based criteria (Zhou et al., 2020; Xu et al., 2020) and 2) linguistic- inspired criteria (Sachan and Xing, 2016; Tay et al., 2019; Nagatsuka et al., …

Content truncated to 500 words