Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11351 - 11361 December 7-11, 2022 ©2022 Association for Computational Linguistics Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models Mengzhou Xia1 Mikel Artetxe2 Jingfei Du2 Danqi Chen1 Ves Stoyanov2 1Princeton University 2Meta AI {mengzhou, danqic}@cs.princeton.edu {artetxe, jingfeidu, ves}@meta.com Abstract Pre-trained masked language models success- fully perform few-shot learning by formulat- ing downstream tasks as text infilling. How- ever, as a strong alternative in full-shot set- tings, discriminative pre-trained models like ELECTRA do not fit into the paradigm. In this work, we adapt prompt-based few-shot learn- ing to ELECTRA and show that it outperforms masked language models in a wide range of tasks. ELECTRA is pre-trained to distinguish if a token is generated or original. We naturally extend that to prompt-based few-shot learning by training to score the originality of the target options without introducing new parameters. Our method can be easily adapted to tasks in- volving multi-token predictions without extra computation overhead. Analysis shows that ELECTRA learns distributions that align better with downstream tasks.1 1 Introduction Large pre-trained language models are known to be effective zero-shot and few-shot learners when scaled (Brown et al., 2020; Artetxe et al., 2021; Rae et al., 2021). Much smaller masked language models (MLMs), like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), can be fined-tuned with only a few examples by utilizing prompt-based fine-tuning, which updates the model to select the correct target word or option (Schick and Schütze, 2021a; Gao et al., 2021). In this paper, we hypothesize that discriminative pre-trained models like ELECTRA (Clark et al., 2020) will make even stronger few-shot learners as alternatives to MLMs as they are pre-trained to dis- tinguish between challenging alternatives. To test this hypothesis, we explore prompt-based learn- ing with ELECTRA by aligning its pre-training 1Code is available at https://github.com/faceb ookresearch/ELECTRA-Fewshot-Learning. Repeat times I am (a) Prompting RoBERTa on SST-2 with single-token options (b) Prompting ELECTRA on SST-2 with single-token options Dis. head replaced original replaced original ∑ c(yi) |y| It was . MLM head great terrible It is pretty damned funny . [MASK] c( ) [MASK] It was . It is pretty damned funny . terrible I ate an apple pie so very hungry now . I am I ate an apple pie so full now . Dis. head c( ) terrible .. It you .. |y| (c) Prompting RoBERTa on COPA with multi-token options Dis. head replaced original (d) Prompting ELECTRA on COPA with multi-token options It was . It is pretty damned funny . great replaced original Dis. head c( ) great [MASK] I ate an apple pie so [MASK][MASK][MASK] . MLM head c( ) [MASK] ∑ c(y′ i) |y′ | Repeat times I am (a) Prompting RoBERTa on SST-2 with single-token options (b) Prompting ELECTRA on SST-2 with single-token options Dis. head replaced original replaced original ∑ c(yi) |y| It was . MLM head great terrible It is pretty damned funny . [MASK] c( ) …

Content truncated to 500 words