TopicBERT for Energy Efficient Document Classification

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1682–1690 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1682 TopicBERT for Energy Efficient Document Classification *Yatin Chaudhary1,2, *Pankaj Gupta1, Khushbu Saxena3, Vivek Kulkarni4, Thomas Runkler3, Hinrich Sch¨utze2 1DRIMCo GmbH, Munich, Germany | 2CIS, University of Munich (LMU), Munich, Germany 3Siemens AG, Munich, Germany | 4Stanford University, California, USA yatin.chaudhary@drimco.net Abstract Prior research notes that BERT’s computa- tional cost grows quadratically with sequence length thus leading to longer training times, higher GPU memory constraints and carbon emissions. While recent work seeks to address these scalability issues at pre-training, these issues are also prominent in fine-tuning espe- cially for long sequence tasks like document classification. Our work thus focuses on op- timizing the computational cost of fine-tuning for document classification. We achieve this by complementary learning of both topic and language models in a unified framework, named TopicBERT. This significantly reduces the number of self-attention operations – a main performance bottleneck. Consequently, our model achieves a 1.4x ( 40%) speedup with  40% reduction in CO2 emission while retaining 99.9% performance over 5 datasets. 1 Introduction Natural Language Processing (NLP) has recently witnessed a series of breakthroughs by the evo- lution of large-scale language models (LM) such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XL- Net (Yang et al., 2019) etc. due to improved capa- bilities for language understanding (Bengio et al., 2003; Mikolov et al., 2013). However this mas- sive increase in model size comes at the expense of very high computational costs: longer train- ing time, high GPU/TPU memory constraints, ad- versely high carbon footprints, and unaffordable invoices for small-scale enterprises. Figure 1 shows the computational cost (training time: millisecond/batch; CO2 emission, and GPU memory usage) of BERT all of which grow quadrat- ically with sequence length (N). We note that this *Equal Contribution Figure 1: Computational cost vs sequence length CO2 BERT pre-training (NAS) (Strubell et al., 2019) 626k BERT fine-training (n=512)* + 125k Table 1: Similar to Strubell et al. (2019) who estimate the carbon footprint of BERT during pretraining, we estimate the carbon footprint (lbs of CO2 equivalent) during finetuning BERT for document classification. *: see supplementary material for details. is primarily due to self-attention operations. More- over, as we note in Table 1, the staggering energy cost is not limited to only the pre-training stage but is also encountered in the fine-tuning stage when processing long sequences as is needed in the task of document classification. Note that the computational cost incurred can be quite signif- icant especially because fine-tuning is more fre- quent than pre-training and BERT is increasingly used for processing long sequences. Therefore, this work focuses on reducing computational cost in the fine-tuning stage of BERT especially for the task of document classification. Recent studies address the excessive computa- tional cost of large language models (LMs) in the pre-training stage using two main compression techniques: (a) Pruning (Michel et al., 2019; …

Content truncated to 500 words