Revisiting k-NN for Fine-tuning Pre-trained Language Models

Revisiting k-NN for Fine-tuning Pre-trained Language Models Lei Li1,2, Jing Chen1,2, Botzhong Tian1,2, Ningyu Zhang1,2∗ 1Zhejiang University & AZFT Joint Lab for Knowledge Engine, China {leili21, chenjing_1984, tbozhong, zhangningyu}@zju.edu.cn Abstract Pre-trained Language Models (PLMs), as parametric-based eager learners, have become the de-facto choice for current paradigms of Natural Language Processing (NLP). In contrast, k- Nearest-Neighbor (k-NN) classifiers, as the lazy learning paradigm, tend to mitigate over-fitting and isolated noise. In this paper, we revisit k-NN classifiers for augmenting the PLMs-based clas- sifiers. From the methodological level, we propose to adopt k-NN with textual representations of PLMs in two steps: (1) Utilize k-NN as prior knowledge to calibrate the training process. (2) Linearly interpolate the probability distribution predicted by k-NN with that of the PLMs’ classifier. At the heart of our approach is the implementation of k-NN-calibrated training, which treats predicted results as indicators for easy versus hard examples during the training process. From the perspective of the diversity of application scenarios, we conduct extensive experiments on fine-tuning, prompt-tuning paradigms and zero-shot, few-shot and fully-supervised settings, respectively, across eight diverse end-tasks. We hope our exploration will encourage the commu- nity to revisit the power of classical methods for efficient NLP1. 1 Introduction eager learners lazy learners Data Training Predicting Output Output Predicting -NN Figure 1: Revisiting how does a lazy learner (k-NN) help the eager learner (PLM). Pre-trained Language Models (PLMs) (Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2020) have shown superior performance across a wide range of language-related downstream tasks (Kowsari et al., 2019; Nan et al., 2020). Afterward, the conventional paradigm fine-tuning, which extends extra task-specific classifiers on the top of PLMs, has been proposed to apply PLMs for downstream tasks. Recently, a new paradigm called prompt-tuning, which originated from GPT-3 (Brown et al., 2020), has been introduced and has shown better results for PLMs on few-shot and zero-shot tasks. Fine- tuning has proved to be effective on supervised tasks and is widely used as the standard method for natural language processing (NLP). Despite the effectiveness of adapting PLMs, parametric-based eager learners (Friedman, 2017), like PLMs with neural networks, require estimating the model parameters ∗ Corresponding Author. 1Code and datasets are available in https://github.com/zjunlp/Revisit-KNN. CCL 2023 Proceedings of the 22nd China National Conference on Computational Linguistics, pages 889-897, Harbin, China, August 3 - 5, 2023. (c) Technical Committee on Computational Linguistics, Chinese Information Processing Society of China 889 with an intensive learning stage. Besides, Training a large PLM model can require significant computing resources and energy, which have negative environmental consequences. As a result, there has been a growing interest in developing more efficient and sustainable methods for training and deploying PLMs. A stark contrast to PLMs is the k-NN classifier: a simplest machine learning algorithm that does not have a training phase but simply predicts labels based on the nearest training examples instead. NLP researchers (Khandelwal et al., 2020; He et al., 2021) have found that k-NN enable excellent uncondi- tional language modeling (Khandelwal …

Content truncated to 500 words