Robust Lottery Tickets for Pre-trained Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2211 - 2224 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Robust Lottery Tickets for Pre-trained Language Models Rui Zheng⋆∗, Rong Bao⋆∗, Yuhao Zhou⋆, Di Liang♠, Sirui Wang♠, Wei Wu♠, Tao Gui♦†, Qi Zhang⋆,♣, Xuanjing Huang⋆ ⋆ School of Computer Science, Fudan University, Shanghai, China ♦ Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China ♣ Shanghai Collaborative Innovation Center of Intelligent Visual Computing, Fudan University ♠ Meituan Inc., Beijing, China {rzheng20,rbao18,tgui,qz,xjhuang}@fudan.edu.cn zhouyh21@m.fudan.edu.cn Abstract Recent works on Lottery Ticket Hypothesis have shown that pre-trained language models (PLMs) contain smaller matching subnetworks (winning tickets) which are capable of reaching accuracy comparable to the original models. However, these tickets are proved to be not robust to adversarial examples, and even worse than their PLM counterparts. To address this problem, we propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original PLMs. Since the loss is not differentiable for the binary mask, we assign the hard concrete distribution to the masks and encourage their sparsity using a smoothing approximation of L0 regularization. Furthermore, we design an adversarial loss objective to guide the search for robust tickets and ensure that the tickets perform well both in accuracy and robustness. Experimental results show the significant improvement of the proposed method over previous work on adversarial robustness evaluation. 1 Introduction Large-scale pre-trained language models (PLMs), such as BERT (Devlin et al., 2019), Roberta (Liu et al., 2019) and T5 (Raffel et al., 2019) have achieved great success in the field of natural language processing. As more transformer layers are stacked with larger self-attention blocks, the complexity of PLMs increases rapidly. Due to the over-parametrization of PLMs, some Transformer heads and even layers can be pruned without significant losses in performance (Michel et al., 2019; Kovaleva et al., 2019; Rogers et al., 2020). The Lottery Ticket Hypothesis suggests an over- parameterized network contains certain subnet- works (i.e., winning tickets) that can match the performance of the original model when trained in isolation (Frankle and Carbin, 2019). Chen ∗ Equal contribution. †Corresponding authors. et al. (2020); Prasanna et al. (2020) also find these winning tickets exist in PLMs. Chen et al. (2020) prune BERT in an unstructured fashion and obtain winning tickets at sparsity from 40% to 90%. Prasanna et al. (2020) aim at finding structurally sparse tickets for BERT by pruning entire attention heads and MLP. Previous works mainly focused on using winning tickets to reduce model size and speed up training time (Chen et al., 2021), while little work has been done to explore more benefits, such as better adversarial robustness than the original model. As we all know, PLMs are vulnerable to adversarial examples that are legitimately crafted by imposing imperceptible perturbations on normal examples (Jin et al., 2020; Garg and Ramakrishnan, 2020; Wang et al., 2021). Recent studies have shown that pruned subnetworks of PLMs are even less …

Content truncated to 500 words