Revisiting Token Dropping Strategy in Efficient BERT Pretraining

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10391–10405 July 9-14, 2023 ©2023 Association for Computational Linguistics Revisiting Token Dropping Strategy in Efficient BERT Pretraining Qihuang Zhong1, Liang Ding2, Juhua Liu3∗, Xuebo Liu4 Min Zhang4, Bo Du1∗, Dacheng Tao5 1 National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China 2 JD Explore Academy, China 3 Research Center for Graphic Communication, Printing and Packaging, and Institute of Artificial Intelligence, Wuhan University, China 4 Institute of Computing and Intelligence, Harbin Institute of Technology, China 5 University of Sydney, Australia {zhongqihuang, liujuhua, dubo}@whu.edu.cn, {liangding.liam, dacheng.tao}@gmail.com Abstract Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the com- putation of a subset of the input tokens at sev- eral middle layers. It can effectively reduce the training time without degrading much per- formance on downstream tasks. However, we empirically find that token dropping is prone to a semantic loss problem and falls short in handling semantic-intense tasks (§2). Moti- vated by this, we propose a simple yet effective semantic-consistent learning method (SCTD) to improve the token dropping. SCTD aims to encourage the model to learn how to preserve the semantic information in the representation space. Extensive experiments on 12 tasks show that, with the help of our SCTD, token drop- ping can achieve consistent and significant per- formance gains across all task types and model sizes. More encouragingly, SCTD saves up to 57% of pretraining time and brings up to +1.56% average improvement over the vanilla token dropping. 1 Introduction Masked language models (MLMs), such as BERT (Devlin et al., 2019) and its variants (Liu et al., 2019; He et al., 2020; Zhong et al., 2023a)1, have achieved great success in a variety of natural language understanding (NLU) tasks. However, with the scaling of model size and corpus size, the pretraining of these BERT-style models becomes more computationally expensive and memory in- tensive (Jiao et al., 2020; Hou et al., 2022). Hence, it is crucial and green to speed up the training and reduce the computational overhead for BERT-style pretraining (Zhang and He, 2020; Schwartz et al., 2020). ∗ Corresponding Authors: Juhua Liu (e-mail: juhua@whu.edu.cn), Bo Du (e-mail: dubo@whu.edu.cn)liu- 1We refer to these models as BERT-style models. Figure 1: Performance of BERTbase on several down- stream tasks. We see that: 1) Despite the remarkable performance on general tasks (i.e., MNLI and SST-2), token dropping leads to dramatically poor performance on the semantic-intense task (i.e., RTE). 2) Our SCTD achieves consistent performance gains among all tasks. To achieve this goal, various training-efficient approaches have been developed and summa- rized (Shoeybi et al., 2019; You et al., 2019; Zhang and He, 2020; Shen et al., 2023). Among these efforts, a recently-proposed token dropping2 strat- egy (Hou et al., 2022) has attracted increasing at- tention owing to its …

Content truncated to 500 words