Task-specific Compression for Multi-task Language Models using Attribution-based Pruning

Findings of the Association for Computational Linguistics: EACL 2023, pages 594–604 May 2-6, 2023 ©2023 Association for Computational Linguistics Task-specific Compression for Multi-task Language Models using Attribution-based Pruning Nakyeong Yang1, Yunah Jang1, Hwanhee Lee2, Seohyeong Jung3 and Kyomin Jung1 1Seoul National University, 2Chung-Ang University, 3Hyundai Motor Group and 42dot Inc {yny0506, vn2209, kjung}@snu.ac.kr hwanheelee@cau.ac.kr, seohyeong.jeong@42dot.ai Abstract Multi-task language models show outstanding performance for various natural language un- derstanding tasks with only a single model. However, these language models utilize an un- necessarily large number of model parameters, even when used only for a specific task. This paper proposes a novel training-free compres- sion method for multi-task language models using a pruning method. Specifically, we use an attribution method to determine which neu- rons are essential for performing a specific task. We task-specifically prune unimportant neurons and leave only task-specific parameters. Fur- thermore, we extend our method to be applica- ble in low-resource and unsupervised settings. Since our compression method is training-free, it uses few computing resources and does not destroy the pre-trained knowledge of language models. Experimental results on the six widely- used datasets show that our proposed prun- ing method significantly outperforms baseline pruning methods. In addition, we demonstrate that our method preserves performance even in an unseen domain setting. 1 Introduction Various pre-trained language models with large- scale data and parameters have emerged (Devlin et al., 2018; Lewis et al., 2019; Raffel et al., 2019; Brown et al., 2020). Specifically, pre-trained lan- guage models like T5 (Raffel et al., 2019) and GPT- 3 (Brown et al., 2020) have shown outstanding performance on many natural language understand- ing tasks. These language models can perform various tasks with a single model by treating ev- ery text processing problem as a text generation problem. However, these language models may uti- lize unnecessary large-scale model parameters even when performing only a specific task. Previous works have introduced various compression meth- ods for language models such as pruning (Chen et al., 2020; Goyal et al., 2020; He et al., 2021), knowledge distillation (Sanh et al., 2019; Hou et al., 2020; Mao et al., 2020; Sun et al., 2020), quan- tization (Shen et al., 2020), and low-rank factor- ization (Liu et al., 2021). However, these studies have (1) not compressed the language models task- specifically or (2) demanded an additional train- ing process like the case of knowledge distillation. This additional training process requires excessive computing resources and a massive training dataset. Furthermore, this training process can destroy in- herent pre-trained knowledge in language models since it updates the model’s pre-trained parameters (Toneva et al., 2018). Due to the catastrophic for- getting (McCloskey and Cohen, 1989) caused by pre-trained knowledge destruction, models which are compressed and trained for a specific task, tend to show degraded performance on solving other pre-trained tasks (Kirkpatrick et al., 2017; Ritter et al., 2018). Also, additional memory space is required to store the trained parameters separately. In this paper, we propose a novel training-free attribution-based task-specific pruning method that …

Content truncated to 500 words