Token and Head Adaptive Transformers for Efficient Natural Language Processing

Proceedings of the 29th International Conference on Computational Linguistics, pages 4575–4584 October 12–17, 2022. 4575 Token and Head Adaptive Transformers for Efficient Natural Language Processing Chonghan Lee1, Md Fahim Faysal Khan1, Rita Brugarolas Brufau2, Ke Ding2, Vijaykrishnan Narayanan1 1The Pennsylvania State University 2Intel 1{cvl5361,mzk591,vxn9}@psu.edu 2{rita.brugarolas.brufau,ke.ding}@intel.com Abstract While pre-trained language models like BERT (Devlin et al., 2019) have achieved im- pressive results on various natural language processing tasks, deploying them on resource- restricted devices is challenging due to their intensive computational cost and memory foot- print. Previous approaches mainly focused on training smaller versions of a BERT model with competitive accuracy under limited com- putational resources. In this paper, we extend Length Adaptive Transformer (Kim and Cho, 2021) and propose to design Token and Head Adaptive Transformer, which can compress and accelerate various BERT-based models via sim- ple fine-tuning. We train a transformer with a progressive token and head pruning scheme, eliminating a large number of redundant tokens and attention heads in the later layers. Then, we conduct a multi-objective evolutionary search with the overall number of floating point oper- ations (FLOPs) as its efficiency constraint to find joint token and head pruning strategies that maximize accuracy and efficiency under vari- ous computational budgets. Empirical studies show that a large portion of tokens and atten- tion heads could be pruned while achieving superior performance compared to the base- line BERT-based models and Length Adaptive Transformers in various downstream NLP tasks. MobileBERT(Sun et al., 2020) trained with our joint token and head pruning scheme achieves a GLUE score of 83.0, which is 1.4 higher than Length Adaptive Transformer and 2.9 higher than the original model. 1 Introduction The Transformer (Vaswani et al., 2017) has be- come the dominant neural architecture used in natural language processing. Especially, pre- trained models using the Transformer architecture as their backbone, e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and GPT-3 (Brown et al., 2020), have shown significant accuracy im- provement across various natural language process- ing tasks. However, the amount of computation required for these large neural networks is enor- mous, which leads to a substantial increase in en- ergy consumption and carbon footprint. For ex- ample, an article points out that in order to train a GPT-3 networks using standard NVIDIA gpus, it would roughly take 190,000 KWh of energy which is equivalent to producing around 85,000 Kg CO2 considering America’s average carbon emission intensity (Katyanna Quach, 2020). Despite their remarkable performance, these power hungry mod- els pose a great hindrance along the pathway to sustainable systems. Hence, it is crucial to make the large-scale networks efficient and sustainable to be deployed on various hardware with different computational budgets and adopted for real-world applications. There have been efforts to improve the efficiency of the large language models (see section 6 for a dis- cussion in detail.) One of the effective approaches has focused on removing redundant model parame- ters. Recent works (Voita et al., 2019; Michel et al., 2019) show that only …

Content truncated to 500 words