Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15746–15761 July 9-14, 2023 ©2023 Association for Computational Linguistics Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models Lijing Wang1∗ Yingya Li2∗ Timothy Miller2 Steven Bethard3 Guergana Savova2 1New Jersey Institute of Technology, lijing.wang@njit.edu 2Boston Children’s Hospital and Harvard Medical School, {firstname.lastname}@childrens.harvard.edu 3 University of Arizona, bethard@email.arizona.edu Abstract The bias-variance tradeoff is the idea that learning methods need to balance model complexity with data size to both under-fitting and over-fitting.minimize Recent empirical work and theoretical analyses with over-parameterized neural networks challenge the classic bias-variance trade-off notion suggesting that no such trade-off holds: as the width of the network grows, bias monotonically decreases while variance initially increases followed by a decrease. In this work, we first provide a variance decomposition-based justification criteria to examine whether large pretrained neural models in a fine-tuning setting are generalizable enough to have low bias and variance. We then perform theoretical and empirical analysis using ensemble methods explicitly designed to decrease variance due to optimization. This results in essentially a two-stage fine-tuning algorithm that first ratchets down bias and variance iteratively, and then uses a selected fixed-bias model to further reduce variance due to optimization by ensembling. We also analyze the nature of variance change with the ensemble size in low- and high-resource classes. Empirical results show that this two-stage method obtains strong results on SuperGLUE tasks and clinical information ex- traction tasks. Code and settings are available: https://github.com/christa60/ bias-var-fine-tuning-plms.git 1 Introduction Transformer-based neural language models, such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), have achieved state-of-the-art (SOTA) performance for a variety of natural language processing (NLP) tasks through the process of fine-tuning. Given an NLP task, the process often involves searching for optimal pretrained models and hyperparameters ∗ co-first authors while continuing to train the pretrained model on a domain-specific dataset, with the aim of building generalizable and robust fine-tuned models. Based on the classic notion of the bias-variance tradeoff (Geman et al., 1992), where increasing model capacity decreases bias but increases vari- ance (leading to a U-shaped test error curve), large pretrained models (LPMs) should have large vari- ance and overfit domain-specific data which is of- ten sparsely labeled and extremely imbalanced for classification. However, recent empirical work and theoretical analysis of neural networks chal- lenge this classic bias-variance trade-off notion (Neal et al., 2018; Yang et al., 2020). It has been suggested that no such trade-off holds: as the width/depth of the network grows, bias monoton- ically decreases while variance initially increases followed by a decrease. This is why transformer- based LPMs often achieve better performance com- pared to less complex models like long short-term memory (LSTM)-based models or feature-rich methods (e.g., support vector machines). In the context of the new bias-variance paradigm, these LPMs are seemingly complex enough to have low bias and variance, however, so far there has been no method to justify whether those SOTA models …

Content truncated to 500 words