Synthesizing Human Gaze Feedback for Improved NLP Performance

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 166–177 May 2-6, 2023 ©2023 Association for Computational Linguistics Do we need Label Regularization to Fine-tune Pre-trained Language Models? Ivan Kobyzev1∗, Aref Jafari1,2∗, Mehdi Rezagholizadeh1, Tianda Li1, Alan Do-Omri1, Peng Lu1,3, Pascal Poupart2, Ali Ghodsi2 1Huawei Noah’s Ark Lab 2 University of Waterloo, Canada 3 Université de Montréal, Canada {ivan.kobyzev,mehdi.rezagholizadeh}@huawei.com {aref.jafari, ppoupart, ali.ghodsi}@uwaterloo.ca Abstract Knowledge Distillation (KD) is a prominent neural model compression technique that heav- ily relies on teacher network predictions to guide the training of a student model. Con- sidering the ever-growing size of pre-trained language models (PLMs), KD is often adopted in many NLP tasks involving PLMs. How- ever, it is evident that in KD, deploying the teacher network during training adds to the memory and computational requirements of training. In the computer vision literature, the necessity of the teacher network is put under scrutiny by showing that KD is a label regu- larization technique that can be replaced with lighter teacher-free variants such as the label- smoothing technique. However, to the best of our knowledge, this issue is not investigated in NLP. Therefore, this work concerns studying different label regularization techniques and whether we actually need them to improve the fine-tuning of smaller PLM networks on down- stream tasks. In this regard, we did a compre- hensive set of experiments on different PLMs such as BERT, RoBERTa, and GPT with more than 600 distinct trials and ran each configu- ration five times. This investigation led to a surprising observation that KD and other la- bel regularization techniques do not play any meaningful role over regular fine-tuning when the student model is pre-trained. We further explore this phenomenon in different settings of NLP and computer vision tasks and demon- strate that pre-training itself acts as a kind of regularization, and additional label regulariza- tion is unnecessary. 1 Introduction Nowadays, we witness the tendency of ever- growing state-of-the-art neural networks. This is especially more evident in natural language pro- cessing (NLP): the famous GPT-3 (Brown et al., 2020) has reached 175 billion parameters and a ∗Equal Contribution Figure 1: DistilRoBERTa results on the test set for the average of seven GLUE tasks. Graph shows the mean performance and one standard deviation interval for the pre-trained and randomly initialized models computed over five runs. For the pre-trained model all intervals intersect, hence label regularization doesn’t improve the performance, but for the model trained from scratch label regularization methods outperform base training. recent Chinese pre-trained language model (Zeng et al., 2021) has 200 billion parameters. It is shown that big over-parameterized neural networks not only have higher VC dimension, and hence more approximation ability (Shalev-Shwartz and Ben- David, 2014), but also their optimization regime is smoother (Safran et al., 2020). At the same time, the optimal point found for big networks has better generalization property (Brutzkus and Globerson, 2019). One can use the advantages of trained big neu- ral networks and transfer their learned …

Content truncated to 500 words