Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1852 - 1865 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency Yanyang Li1∗, Fuli Luo2, Runxin Xu3, Songfang Huang2, Fei Huang2, Liwei Wang1 1Department of Computer Science and Engineering, The Chinese University of Hong Kong 2Alibaba Group 3Key Laboratory of Computational Linguistics, Peking University, MOE, China {yyli21,lwwang}@cse.cuhk.edu.hk, runxinxu@gmail.com {lfl259702,songfang.hsf,f.huang}@alibaba-inc.com Abstract Structured pruning has been extensively stud- ied on monolingual pre-trained language mod- els and is yet to be fully evaluated on their mul- tilingual counterparts. This work investigates three aspects of structured pruning on multi- lingual pre-trained language models: settings, algorithms, and efficiency. Experiments on nine downstream tasks show several counter- intuitive phenomena: for settings, individu- ally pruning for each language does not in- duce a better result; for algorithms, the sim- plest method performs the best; for efficiency, a fast model does not imply that it is also small. To facilitate the comparison on all sparsity lev- els, we present Dynamic Sparsification, a sim- ple approach that allows training the model once and adapting to different model sizes at inference. We hope this work fills the gap in the study of structured pruning on multilingual pre-trained models and sheds light on future re- search. 1 Introduction Large-scale pre-trained monolingual language mod- els like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown promising results in various NLP tasks while suffering from their large model size and high latency. Structured pruning has proven to be an effective approach to compressing and accelerating these large monolingual language models (Michel et al., 2019; Wang et al., 2020c; Prasanna et al., 2020; Liang et al., 2021), making them practical for real-world applications. Similarly, multilingual pre-trained models (Con- neau and Lample, 2019; Conneau et al., 2020; Xue et al., 2021; Luo et al., 2021) are also powerful and even have more parameters. However, little atten- tion has been paid to evaluating the effectiveness of structured pruning on these multilingual mod- els. Applying pruning to multilingual pre-trained ∗ Collaborated work while doing an Alibaba DAMO Academy internship. models is non-trivial, as it typically involves many languages and needs to carefully design the roles of modules within the network. For example, most at- tention heads have little impact on the performance of monolingual pre-trained models (Michel et al., 2019; Voita et al., 2019), while it is the opposite for multilingual pre-trained models (See Section 5.3 and also Budhraja et al. (2021)). This work intends to examine how structured pruning reacts to multilingual pre-trained mod- els. We take the most representative multilingual pre-trained model family, XLM-R (Conneau et al., 2020; Goyal et al., 2021) for our case study and evaluate the pruning performance on nine cross- lingual understanding tasks in XTREME (Hu et al., 2020). We investigate three aspects of structured pruning: settings, algorithms, and efficiency. Settings Traditional pruning produces a single small model, which …

Content truncated to 500 words