Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 225–235 July 13-14, 2023 ©2023 Association for Computational Linguistics Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learning Hyunsoo Cho†, Choonghyun Park†, Junyeop Kim†, Hyuhng Joon Kim†, Kang Min Yoo†‡, Sang-goo Lee† † Seoul National University, ‡ NAVER {johyunsoo,pch330,juny116,heyjoonkim,sglee}@europa.snu.ac.kr {kangmin.yoo}@navercorp.com Abstract As the size of the pre-trained language model (PLM) continues to increase, numerous parameter-efficient transfer learning methods have been proposed recently to compensate for the tremendous cost of fine-tuning. Despite the impressive results achieved by large pre- trained language models (PLMs) and various parameter-efficient transfer learning (PETL) methods on sundry benchmarks, it remains un- clear if they can handle inputs that have been distributionally shifted effectively. In this study, we systematically explore how the ability to de- tect out-of-distribution (OOD) changes as the size of the PLM grows or the transfer meth- ods are altered. Specifically, we evaluated var- ious PETL techniques, including fine-tuning, Adapter, LoRA, and prefix-tuning, on three dif- ferent intention classification tasks, each uti- lizing various language models with different scales. 1 Introduction Pre-trained language models (PLM), which are pre- trained on large-scale corpora using transformer- based architectures (Vaswani et al., 2017), have achieved groundbreaking success on sundry bench- marks (Wang et al., 2019b; Rajpurkar et al., 2016; Wang et al., 2019a), establishing themselves as the standard neural model in countless applications. Moreover, language models pre-trained with larger parameters on a rich volume of corpora tend to ex- hibit more intriguing potentials, such as the ability to capture world knowledge (Petroni et al., 2019), generate codes (Poesia et al., 2022), and even solve mathematical problems (Henighan et al., 2020), on top of understanding linguistic knowledge (e.g., semantic or syntactic). To explore the apex of pre- trained language models (PLMs), the size of PLMs is growing exponentially and has reached billions to a trillion (Brown et al., 2020; Chowdhery et al., 2022; Fedus et al., 2022; Hoffmann et al., 2022). Under these circumstances, the conventional method for transferring PLMs to a target task (i.e., fine-tuning) is now infeasible as it entails prohibitive costs to train and store the entire pa- rameters of large PLMs for every desired task. To mitigate this issue, several recent parameter- efficient transfer learning (PETL) methods have been proposed to improve task scalability. For in- stance, adapter-based (Houlsby et al., 2019; Hu et al., 2022) approaches insert small neural mod- ules into each layer of the PLM and update those lightweight modules in the training phase. Inspired by the recent success of textual prompts (Brown et al., 2020), prompt-based methods (Li and Liang, 2021; Lester et al., 2021; Shin et al., 2020) concate- nate extra tunable tokens to the front of the input or hidden layers and update prepended soft prompts in the training phase. Despite these breakthroughs in NLP, even very recent anomaly detection studies (Cho et al., 2022; Shen et al., 2021) are still limited to relatively small bi-directional PLMs (e.g., BERT, RoBERTa). Thus, how …

Content truncated to 500 words