Dynamic Transformers Provide a False Sense of Efficiency

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7164–7180 July 9-14, 2023 ©2023 Association for Computational Linguistics Dynamic Transformers Provide a False Sense of Efficiency Yiming Chen† Simin Chen‡∗ Zexin Li⋆ Wei Yang‡ Cong Liu⋆ Robby T. Tan† Haizhou Li♮,†,§ †National University of Singapore ⋆University of California, Riverside♮The Chinese University of Hong Kong, Shenzhen ‡University of Texas at Dallas §Kriston AI Lab yiming.chen@u.nus.edu, {simin.chen,wei.yang}@utdallas.edu {zli536,congl}@ucr.edu, {robby.tan,haizhou.li}@nus.edu.sg Abstract Despite much success in natural language pro- cessing (NLP), pre-trained language models typically lead to a high computational cost dur- ing inference. Multi-exit is a mainstream ap- proach to address this issue by making a trade- off between efficiency and accuracy, where the saving of computation comes from an early exit. However, whether such saving from early- exiting is robust remains unknown. Motivated by this, we first show that directly adapting ex- isting adversarial attack approaches targeting model accuracy cannot significantly reduce in- ference efficiency. To this end, we propose a simple yet effective attacking framework, SAME, a novel slowdown attack framework on multi-exit models, which is specially tai- lored to reduce the efficiency of the multi-exit models. By leveraging the multi-exit models’ design characteristics, we utilize all internal predictions to guide the adversarial sample gen- eration instead of merely considering the final prediction. Experiments on the GLUE bench- mark show that SAME can effectively diminish the efficiency gain of various multi-exit models by 80% on average, convincingly validating its effectiveness and generalization ability. 1 1 Introduction Pre-trained language models (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Chen et al., 2022b) have shown great potential in a wide range of NLP tasks. While large language models offer unparalleled per- formance, their high computation during inference limits the scope of applications. More studies re- cently concentrate on efficient NLP, which aims to speed up the inference of deep language models without significant performance degradation (Sanh et al., 2019; Zafrir et al., 2019; Zhou et al., 2020). Among these, the multi-exit models (Zhou et al., 2020; Xin et al., 2020) attract widespread attention. ∗ Corresponding author. 1Code is available at github.com/MatthewCYM/SAME The idea of the multi-exit models stems from the observation that inputs with varying seman- tics demand distinct computational resources. By automatically adjusting different computational re- sources according to input semantics, one can ef- fectively speed up the inference of a multi-exit model with minimum performance loss. Further- more, such multi-exit model can be easily com- bined with other static speedup approaches, e.g., distillation (Sanh et al., 2019; Jiao et al., 2020), by replacing the backbone model. In addition to higher efficiency, previous studies also show that the multi- exit models are more robust to correctness-based adversarial samples (Zhou et al., 2020; Hu et al., 2020). The study of NLP attacks has mostly focused on harming models’ accuracy, and taken static trans- formers as victim models (Ebrahimi et al., 2018b; Li et …

Content truncated to 500 words