Analyzing Redundancy in Pretrained Transformer Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4908–4926, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4908 Analyzing Redundancy in Pretrained Transformer Models Fahim Dalvi Hassan Sajjad Nadir Durrani Yonatan Belinkov* {faimaduddin,hsajjad,ndurrani}@hbku.edu.qa Qatar Computing Research Institute, HBKU Research Complex, Doha 5825, Qatar *MIT Computer Science and Artificial Intelligence Laboratory and Harvard John A. Paulson School of Engineering and Applied Sciences, Cambridge, MA, USA belinkov@csail.mit.edu Abstract Transformer-based deep NLP models are trained using hundreds of millions of param- eters, limiting their applicability in computa- tionally constrained environments. In this pa- per, we study the cause of these limitations by defining a notion of Redundancy, which we categorize into two classes: General Re- dundancy and Task-specific Redundancy. We dissect two popular pretrained models, BERT and XLNet, studying how much redundancy they exhibit at a representation-level and at a more fine-grained neuron-level. Our anal- ysis reveals interesting insights, such as: i) 85% of the neurons across the network are redundant and ii) at least 92% of them can be removed when optimizing towards a down- stream task. Based on our analysis, we present an efficient feature-based transfer learning pro- cedure, which maintains 97% performance while using at-most 10% of the original neu- rons.1 1 Introduction Large pretrained models have improved the state- of-the-art in a variety of NLP tasks, with each new model introducing deeper and wider architectures causing a significant increase in the number of pa- rameters. For example, BERT large (Devlin et al., 2019), NVIDIA’s Megatron model, and Google’s T5 model (Raffel et al., 2019) were trained using 340 million, 8.3 billion and 11 billion parameters respectively. An emerging body of work shows that these mod- els are over-parameterized and do not require all the representational power lent by the rich archi- tectural choices during inference. For example, these models can be distilled (Sanh et al., 2019; at1The code for the experiments in this paper is available https://github.com/fdalvi/analyzing- redundancy-in-pretrained-transformer- models Sun et al., 2019) or pruned (Voita et al., 2019; Sajjad et al., 2020), with a minor drop in perfor- mance. Recent research (Mu et al., 2018; Etha- yarajh, 2019) analyzed contextualized embeddings in pretrained models and showed that the repre- sentations learned within these models are highly anisotropic. While these approaches successfully exploited over-parameterization and redundancy in pretrained models, the choice of what to prune is empirically motivated and the work does not directly explore the redundancy in the network. Identifying and analyzing redundant parts of the network is useful in: i) developing a better under- standing of these models, ii) guiding research on compact and efficient models, and iii) leading to- wards better architectural choices. In this paper, we analyze redundancy in pre- trained models. We classify it into general redun- dancy and task-specific redundancy. The former is defined as the redundant information present in a pretrained model irrespective of any down- stream task. This redundancy is an artifact of over- parameterization and other training choices that force various parts of the models …

Content truncated to 500 words