Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 1788–1805 May 2-6, 2023 ©2023 Association for Computational Linguistics Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models Mohammadreza Banaei∗1, Klaudia Bałazy∗2, Artur Kasymov2 Rémi Lebret1, Jacek Tabor2, Karl Aberer1 1EPFL, 2Jagiellonian University Abstract Recent transformer language models achieve outstanding results in many natural language processing (NLP) tasks. However, their enor- mous size often makes them impractical on memory-constrained devices, requiring practi- tioners to compress them to smaller networks. In this paper, we explore offline compres- sion methods, meaning computationally-cheap approaches that do not require further fine- tuning of the compressed model. We chal- lenge the classical matrix factorization meth- ods by proposing a novel, better-performing autoencoder-based framework. We perform a comprehensive ablation study of our approach, examining its different aspects over a diverse set of evaluation settings. Moreover, we show that enabling collaboration between modules across layers by compressing certain modules together positively impacts the final model per- formance. Experiments on various NLP tasks demonstrate that our approach significantly out- performs commonly used factorization-based offline compression methods.1. 1 Introduction The recent trend of pre-training Trans- former (Vaswani et al., 2017) language models on enormous unsupervised corpus has led to outstanding performances on many downstream tasks. For downstream tasks, these pre-trained models are then either fine-tuned (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019) or the prompting paradigm (Brown et al., 2020) is used (especially in the so-called Large Language Models), which avoids having a different model per task (Lieber et al., 2021; Rae et al., 2021; Smith et al., 2022; Thoppilan et al., 2022). In each of the two paradigms, it has been shown *Equal contribution Correspondence to: mohammadreza.banaei@epfl.ch, klaudia.balazy@doctoral.uj.edu.pl 1Our code is public: github.com/MohammadrezaBanaei/auto- encoder-based-transformer-compression that increasing the scale of language models generally leads to better performance on a range of downstream tasks (Devlin et al., 2019; Brown et al., 2020). Indeed, for autoregressive language models, Kaplan et al. (2020) demonstrated a power-law relationship between the number of parameters and the respective performance. Wei et al. (2022) further showed that certain abilities of language models emerge only when the number of its parameters passes certain thresholds, providing an incentive to scale these models further. Although scaling up these language models make them empirically powerful across many di- verse tasks, it makes them infeasible to train for many NLP practitioners due to huge pre-training costs. More importantly, even using the available pre-trained models for inference is becoming more challenging (especially for memory-constrained ap- plications like edge devices), with recent models having hundreds of billions of parameters (Zhang et al., 2022). With the rise of NLP model sizes, there have been many efforts to compress transformer lan- guage models without compromising their perfor- mance. Although being inherently different, many of these efforts rely on knowledge distillation (Hin- ton et al., 2015) to help the compressed model (i.e., the student model) better imitate the parent model (i.e., the teacher model). However, these ap- proaches often …

Content truncated to 500 words