Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7042 - 7055 December 7-11, 2022 ©2022 Association for Computational Linguistics Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective Ping Yang1∗ Junjie Wang2∗ Ruyi Gan1 Xinyu Zhu3 Lin Zhang1 Ziwei Wu1 Xinyu Gao1 Jiaxing Zhang1 Tetsuya Sakai2† 1International Digital Economy Academy 2Waseda University 3Tsinghua University {yangping, ganruyi, zhanglin, wuziwei, gaoxinyu, zhangjiaxing}@idea.edu.cn wjj1020181822@toki.waseda.jp tetsuyasakai@acm.org zhuxy21@mails.tsinghua.edu.cn Abstract We propose a new paradigm for zero-shot learners that is format agnostic, i.e., it is compatible with any format and applicable to a list of language tasks, such as text classifi- cation, commonsense reasoning, coreference resolution, and sentiment analysis. Zero-shot learning aims to train a model on a given task such that it can address new learning tasks without any additional training. Our approach converts zero-shot learning into multiple-choice tasks, avoiding problems in commonly used large-scale generative models such as FLAN. It not only adds generalization ability to models but also significantly reduces the number of parameters. Our method shares the merits of efficient training and deployment. Our approach shows state-of-the-art perfor- mance on several benchmarks and produces satisfactory results on tasks such as natural language inference and text classification. Our model achieves this success with only 235M parameters, which is substantially smaller than state-of-the-art models with billions of parameters. The code and pre-trained models are available at https://github. com/IDEA-CCNL/Fengshenbang-LM/tree/ main/fengshen/examples/unimc. 1 Introduction Remarkable advances in large-scale language mod- els have brought substantial improvements in a wide variety of tasks such as text classification, natural language inference and commonsense rea- soning (Brown et al., 2020; Chowdhery et al., 2022). This progress brings opportunities to Zero- Shot Learning (ZSL) (Sanh et al., 2021; Chowdh- ery et al., 2022), which aims to predict labels on datasets from novel domains. Most solutions can be framed in the prompt tuning framework that ac- tivate specific parameters in PLM (Xu et al., 2022; *Equal contribution. †Corresponding Author. PMLM PLM (c) Fine tuning (BERT) Fine-tune on task A Inference on task A 🗙 No task-specific examples Classifier: Random initialization (a) Prompt tuning (T5, GPT) Inference on unseen task A Prompt engineering Verbalizer design PLM (b) Instruction tuning (FLAN) Inference on unseen task A Instruction-tune on other task clusters (B,C,D,E, …) Template design Clustering PMLM (d) Multiple Choice tuning (UniMC) Inference on unseen task A, B, C, … MC training (Only MC task) O-MLM, OP MLM, O-MLM, OP Figure 1: Typical zero-shot learning methods and our proposed UniMC. “PLM” indicates pre-trained lan- guage model. “PMLM” implies pre-trained masked language model. T0 11B GLaM 60B FLAN 137B PaLM 540B UniMC 235M (ours) 35.0 40.0 45.0 50.0 55.0 1.0 10.0 100.0 1000.0 10000.0 Accuracy (%) Model size (# of paramets in log scale) Figure 2: Zero-shot performance comparison in ALNI R1. Our proposed UniMC has the best performance w.r.t the accuracy and the model size, simultaneously. Liu et al., 2021) to adapt to zero-shot tasks. A pow- erful variant of prompt tuning is called instruction 7042 tuning …

Content truncated to 500 words