Few-Shot Learning with Siamese Networks and Label Tuning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8532 - 8545 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Few-Shot Learning with Siamese Networks and Label Tuning Thomas M¨uller and Guillermo P´erez-Torr´o and Marc Franco-Salvador Symanto Research, Valencia, Spain https://www.symanto.com {thomas.mueller,guillermo.perez,marc.franco}@symanto.com Abstract We study the problem of building text classi- fiers with little or no training data, commonly known as zero and few-shot text classification. In recent years, an approach based on neu- ral textual entailment models has been found to give strong results on a diverse range of tasks. In this work, we show that with proper pre-training, Siamese Networks that embed texts and labels offer a competitive alternative. These models allow for a large reduction in in- ference cost: constant in the number of labels rather than linear. Furthermore, we introduce label tuning, a simple and computationally effi- cient approach that allows to adapt the models in a few-shot setup by only changing the label embeddings. While giving lower performance than model fine-tuning, this approach has the architectural advantage that a single encoder can be shared by many different tasks. 1 Introduction Few-shot learning is the problem of learning classi- fiers with only a few training examples. Zero-shot learning (Larochelle et al., 2008), also known as dataless classification (Chang et al., 2008), is the extreme case, in which no labeled data is used. For text data, this is usually accomplished by represent- ing the labels of the task in a textual form, which can either be the name of the label or a concise textual description. In recent years, there has been a surge in zero- shot and few-shot approaches to text classification. One approach (Yin et al., 2019, 2020; Halder et al., 2020; Wang et al., 2021) makes use of entailment models. Textual entailment (Dagan et al., 2006), also known as natural language inference (NLI) (Bowman et al., 2015), is the problem of predicting whether a textual premise implies a textual hypoth- esis in a logical sense. For example, Emma loves apples implies that Emma likes apples. The entailment approach for text classification sets the input text as the premise and the text repre- senting the label as the hypothesis. A NLI model is applied to each input pair and the entailment probability is used to identify the best matching label. In this paper, we investigate an alternative based on Siamese Networks (SN) (Bromley et al., 1993), also known as dual encoders. These models embed both input and label texts into a common vector space. The similarity of the two items can then be computed using a similarity function such as the dot product. The advantage is that input and label text are encoded independently, which means that the label embeddings can be pre-computed. There- fore, at inference time, only a single call to the model per input is needed. In contrast, the models typically applied in the entailment approach are Cross Attention (CA) models which need to …

Content truncated to 500 words