Embedding Hallucination for Few-shot Language Fine-tuning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5522 - 5530 July 10-15, 2022 ©2022 Association for Computational Linguistics Embedding Hallucination for Few-Shot Language Fine-tuning Yiren Jian∗ Dartmouth College yiren.jian.gr@dartmouth.edu Chongyang Gao ∗ Northwestern University cygao@u.northwestern.edu Soroush Vosoughi Dartmouth College soroush@dartmouth.edu Abstract Few-shot language learners adapt knowledge from a pre-trained model to recognize novel classes from a few-labeled sentences. In such settings, fine-tuning a pre-trained language model can cause severe over-fitting. In this pa- per, we propose an Embedding Hallucination (EmbedHalluc) method, which generates auxil- iary embedding-label pairs to expand the fine- tuning dataset. The hallucinator is trained by playing an adversarial game with the discrim- inator, such that the hallucinated embedding is indiscriminative to the real ones in the fine- tuning dataset. By training with the extended dataset, the language learner effectively learns from the diverse hallucinated embeddings to overcome the over-fitting issue. Experiments demonstrate that our proposed method is ef- fective in a wide range of language tasks, out- performing current fine-tuning methods. Fur- ther, we show that EmbedHalluc outperforms other methods that address this over-fitting problem, such as common data augmentation, semi-supervised pseudo-labeling, and regular- ization. The code will be made available at: https://github.com/yiren-jian/EmbedHalluc. 1 Introduction Fine-tuning a pre-trained language model (LM) on a downstream task with the labeled data has been the de facto approach in many NLP tasks (Wang et al., 2019; Devlin et al., 2019). Conventional fine- tuning has been shown to be effective when a few thousands of labeled examples are available. Data augmentation (Wei and Zou, 2019), regularization (Lee et al., 2019) and re-initialization (Zhang et al., 2021) further improve the results. However, the performance drops drastically when the number of examples falls to only a few dozens. Experiments from recent work (Gao et al., 2021) have shown that fine-tuning performs poorly in the setting where only 16 examples per class are ∗Both authors contributed equally to this research. given. Indeed, tuning a language model with hun- dreds of millions of parameters (e.g., BERT-large has 300M parameters) with only a few examples inevitably faces the over-fitting problem. Prior work have proposed regularization meth- ods to overcome this problem (Lee et al., 2019; Zhang et al., 2021). However, we show in our ex- periments that these methods fail in extreme data scarce setting. We speculate that the key to solve this issue is by data augmentation. Current common text data augmentation meth- ods, such as EDA (Wei and Zou, 2019) (which have been used in recent few-shot learning papers (Wei et al., 2021; Basu et al., 2021)) and AEDA (Karimi et al., 2021) operate at the lexical level, which while resulting in human readable texts, lead to limited diversity due to the discrete nature of the lexical space. In this work, we propose to use a generative augmentation method at the embedding space for few-shot learning. The underlying hypothesis is that the intra-class relation of the observed exam- ples can be …

Content truncated to 500 words