How to Train BERT with an Academic Budget

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10644–10652 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10644 How to Train BERT with an Academic Budget Peter Izsak♣ Moshe Berchansky♣ Omer Levy♦ ♣Intel Labs, Israel ♦Blavatnik School of Computer Science, Tel Aviv University {peter.izsak,moshe.berchansky}@intel.com levyomer@cs.tau.ac.il Abstract While large language models à la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well- funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low- end deep learning server. We demonstrate that through a combination of software optimiza- tions, design choices, and hyperparameter tun- ing, it is possible to produce models that are competitive with BERTBASE on GLUE tasks at a fraction of the original pretraining cost.1 1 Introduction Large language models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and GPT3 (Brown et al., 2020), have become the de facto models used in many NLP tasks. However, their pretraining phase can be prohibitively expensive for startups and academic research groups, limiting the research and development of model pretraining to only a few well-funded industry labs. How can one train a large language model with commonly- available hardware in reasonable time? We present a recipe for training a BERT-like masked language model (MLM) in 24 hours in a limited computation environment. Our approach combines multiple elements from recent work: faster implementation (Rasley et al., 2020), faster convergence through over-parameterization (Li et al., 2020b), best practices for scaling language models (Kaplan et al., 2020), single-sequence train- ing (Joshi et al., 2020; Liu et al., 2019), and more. Moreover, we conduct an extensive hyperparam- eter search tailored to our resource budget, and find that synchronizing learning rate warmup and decay schedules with our 24 hour budget greatly improves model performance. 1Our code is publicly available at: https://github. com/IntelLabs/academic-budget-bert When evaluating on GLUE (Wang et al., 2018), our recipe produces models that are competitive with BERTBASE – a model that was trained on 16 TPUs for 4 days. This recipe can also be applied to other corpora, as we demonstrate by training a French-language model on par with CamemBERTBASE (Martin et al., 2020) on the XNLI French benchmark (Conneau et al., 2018). Overall, our findings demonstrate that, with the right recipe and an understanding of the available computational resources, large language models can indeed be trained in an academic setting. 2 Problem Setup We investigate the task of pretraining a large lan- guage model under computational constraints. To simulate an academic computation budget, we limit the training time to 24 hours and the hardware to a single low-end deep learning server.2 Using current cloud-compute prices, we estimate the dollar-cost of each training run at around $50 to $100. Under these constraints, our goal is to pretrain a model that can benefit classification tasks, such as in GLUE …

Content truncated to 500 words