Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4799–4816 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations Linlin Liu1,2∗ Xingxuan Li1,2∗ Megh Thakkar2 Xin Li1 Shafiq Joty2,3 Luo Si1 Lidong Bing1† 1DAMO Academy, Alibaba Group 2Nanyang Technological University 3Salesforce AI {linlin.liu, xingxuan.li, xinting.lx, luo.si, l.bing}@alibaba-inc.com megh.1211@gmail.com srjoty@ntu.edu.sg Abstract Due to the huge amount of parameters, fine- tuning of pretrained language models (PLMs) is prone to overfitting in the low resource sce- narios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders be- tween the hidden layers of a PLM, which trans- form activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoen- coders are plugged out after fine-tuning, so our method does not add extra parameters or in- crease computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequence- and token-level low-resource NLP tasks. Our code is available at https://github.com/DAMO- NLP-SG/MVCR. 1 Introduction Fine-tuning pretrained language models (PLMs) (Devlin et al., 2019; Conneau and Lample, 2019; Liu et al., 2020) provides an efficient way to trans- fer knowledge gained from large scale text corpora to downstream NLP tasks, which has achieved state-of-the-art performance on a wide range of tasks (Yang et al., 2019; Yamada et al., 2020; Chi et al., 2021; Sun et al., 2021; Zhang et al., 2021b,a; Chia et al., 2022; Tan et al., 2022; Zhou et al., 2022b). However, most of the PLMs are designed for general purpose representation learning (Liu et al., 2019; Conneau et al., 2020), so the learned representations unavoidably contain abundant fea- tures irrelevant to the downstream tasks. Moreover, the PLMs typically possess a huge amount of pa- rameters (often 100+ millions) (Brown et al., 2020; ∗Equal contribution, order decided by coin flip. Linlin Liu and Xingxuan Li are under the Joint Ph.D. Program between Alibaba and Nanyang Technological University. †Corresponding author. Min et al., 2021), which makes them more expres- sive compared with simple models, and hence more vulnerable to overfitting noise or irrelevant features during fine-tuning, especially in the low-resource scenarios. There has been a long line of research on devis- ing methods to prevent large neural models from overfitting. The most common ones can be roughly grouped into three main categories: data augmenta- tion (DeVries and Taylor, 2017; Ding et al., 2020; Liu et al., 2021; Feng et al., 2021; Xu et al., 2023; Zhou et al., 2022c), parameter/activation regular- ization (Krogh and Hertz, 1991; Srivastava et al., 2014; Lee et al., 2020; Li et al., 2020) and label smoothing (Szegedy et al., 2016; Yuan et al., 2020), which, from bottom to top, operates on data sam- ples, model parameters/activations and data labels, respectively. Data augmentation methods, such as back- translation (Sennrich et al., 2016) and masked pre- diction (Bari et …

Content truncated to 500 words