Efficient Adversarial Training with Robust Early-Bird Tickets

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8318 - 8331 December 7-11, 2022 ©2022 Association for Computational Linguistics Efficient Adversarial Training with Robust Early-Bird Tickets Zhiheng Xi1∗, Rui Zheng1∗, Tao Gui2† , Qi Zhang1, Xuanjing Huang1,3 1 School of Computer Science, Fudan University, Shanghai, China 2 Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China 3 International Human Phenome Institutes (Shanghai), Shanghai, China zhxi22@m.fudan.edu.cn , {rzheng20,tgui,qz,xjhuang}@fudan.edu.cn Abstract Adversarial training is one of the most powerful methods to improve the robustness of pre- trained language models (PLMs). However, this approach is typically more expensive than traditional fine-tuning because of the necessity to generate adversarial examples via gradient descent. Delving into the optimization process of adversarial training, we find that robust connectivity patterns emerge in the early training phase (typically 0.15 ∼ 0.3 epochs), far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets (i.e., subnetworks) to develop an efficient adversarial training method: (1) searching for robust tickets with structured sparsity in the early stage; (2) fine-tuning robust tickets in the remaining time. To extract the robust tickets as early as possible, we design a ticket convergence metric to automatically terminate the searching process. Experiments show that the proposed efficient adversarial training method can achieve up to 7× ∼ 13× training speedups while maintaining comparable or even better robustness compared to the most competitive state-of-the-art adversarial training methods. 1 Introduction Pre-trained language models (PLMs) have achieved great success in NLP (Devlin et al., 2019a), but they are vulnerable to adversarial examples crafted by performing subtle perturbations on normal examples (Ren et al., 2019; Garg and Ramakrishnan, 2020). Recent studies have shown the prevalence of adversarial vulnerability in NLP tasks (Wallace et al., 2019; Zhang et al., 2021; Lin et al., 2021). Various defense strategies have been proposed to improve the robustness of the model and maintain high accuracy on both normal and adversarial examples (Zhu et al., 2020; Wang et al., ∗Equal contribution. †Corresponding author. Adversarial Training Methods H2 H1 H12 Traditional Fine-tuning EarlyRobust (Ours) Model Optimization Adversarial Training ... ... FFN MHA H2 H1 H12 ... ... Structured Pruning 100% Adversarial Training Adversarial Examples Generation 6 10 Fine-tuning Note: Training Time Comparison 1.5% ~ 3.0% Adversarial Training Figure 1: A high-level overview of adversarial training methods and the proposed EarlyRobust. MHA means multi-head attention and FFN means feed-forward network. Structured pruning removes attention heads and intermediate neurons of each transformer layer in BERT. Our method achieves up to 7× ∼ 13× training speedups compared to adversarial training methods. 2021). Adversarial training is one of the most widely used and strongest defense methods (Madry et al., 2018). One of the main challenges of adversarial training in real-world applications is its high com- putational cost, as they require multi-step gradient descents to generate adversarial examples (Wong et al., 2020; Andriushchenko and Flammarion, 2020). Some work on this topic found that replacing strong adversarial examples with weaker and cheaper ones does not significantly …

Content truncated to 500 words