Distinguishability Calibration to In-Context Learning

Findings of the Association for Computational Linguistics: EACL 2023, pages 1385–1397 May 2-6, 2023 ©2023 Association for Computational Linguistics Distinguishability Calibration to In-Context Learning Hongjing Li1∗, Hanqi Yan1∗, Yanran Li, Li Qian2, Yulan He1,3,4, and Lin Gui3 1Department of Computer Science, University of Warwick, UK 2Xiaomi AI Lab, China 3Department of informatics, King’s College London, UK 4The Alan Turing Institute, UK {Hongjing.Li,Hanqi.Yan}@warwick.ac.uk, yanranli.summer@gmail.com, qianli@xiaomi.com, {yulan.he,lin.1.gui}@kcl.ac.uk Abstract Recent years have witnessed increasing inter- ests in prompt-based learning in which mod- els can be trained on only a few annotated in- stances, making them suitable in low-resource settings. When using prompt-based learning for text classification, the goal is to use a pre- trained language model (PLM) to predict a missing token in a pre-defined template given an input text, which can be mapped to a class label. However, PLMs built on the transformer architecture tend to generate similar output em- beddings, making it difficult to discriminate between different class labels. The problem is further exacerbated when dealing with classifi- cation tasks involving many fine-grained class labels. In this work, we alleviate this informa- tion diffusion issue, i.e., different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer, by proposing a calibra- tion method built on feature transformations through rotation and scaling to map a PLM- encoded embedding into a new metric space to guarantee the distinguishability of the resulting embeddings. Furthermore, we take the advan- tage of hyperbolic embeddings to capture the hierarchical relations among fine-grained class- associated token embedding by a coarse-to-fine metric learning strategy to enhance the distin- guishability of the learned output embeddings. Extensive experiments on the three datasets un- der various settings demonstrate the effective- ness of our approach. 1 1 Introduction Large pre-trained language models (PLMs) (Devlin et al., 2019; Lan et al., 2020; Liu et al., 2019) have been achieved state-of-the-art performance in many Natural Language Processing (NLP) downstream tasks. More recently, the PLMs with prompt learn- ing demonstrate surprising capabilities in numerous *Equal contribution. 1Our code can be found at https://github.com/dontt al/TARA tasks both in NLP and computer vision, even out- performing their fine-tuned counterparts (Brown et al., 2020; Liu et al., 2021; Lester et al., 2021; Zhou et al., 2022b; Gao et al., 2021a). Train#1: Gotta protect’em! It was [MASK]. Train#2: That’s why it’s only 20$. It was [MASK]. Test: On a boat trip to Denmark. It was [MASK]. Table 1: The prompt templates for emotion classifica- tion. The samples are from GoEmotion (Demszky et al., 2020) dataset. In an emotion classification task shown in Table 1, an input sentence X, followed by a prompt, “It was [MASK]”, is fed to a PLM to predict the miss- ing token at the position of [MASK]. The predicted word can be used to identify the emotion label of the input sentence. Such few-shot learning gen- erates a probability distribution over the [MASK] conditioning on the given prompt/context, which is considered as in-context learning of language models. However, …

Content truncated to 500 words