ConEntail: An Entailment-based Framework for Universal Zero and Few Shot Classification with Supervised Contrastive Pretraining

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1941–1953 May 2-6, 2023 ©2023 Association for Computational Linguistics CONENTAIL: An Entailment-based Framework for Universal Zero and Few Shot Classification with Supervised Contrastive Pretraining Ranran Haoran Zhang1, Aysa Xuemo Fan2, Rui Zhang1 1 Penn State University 2 University of Illinois at Urbana-Champaign {hzz5361, rmz5227}@psu.edu xuemof2@illinois.edu Abstract A universal classification model aims to gener- alize to diverse classification tasks in both zero and few shot settings. A promising way toward universal classification is to cast heterogeneous data formats into a dataset-agnostic “meta-task” (e.g., textual entailment, question answering) then pretrain a model on the combined meta dataset. The existing work is either pretrained on specific subsets of classification tasks, or pretrained on both classification and genera- tion data but the model could not fulfill its po- tential in universality and reliability. These also leave a massive amount of annotated data under-exploited. To fill these gaps, we propose CONENTAIL, a new framework for universal zero and few shot classification with supervised contrastive pretraining. Our unified meta-task for classification is based on nested entailment. It can be interpreted as “Does sentence a en- tails [sentence b entails label c]”. This formu- lation enables us to make better use of 57 an- notated classification datasets for supervised contrastive pretraining and universal evaluation. In this way, CONENTAIL helps the model (1) absorb knowledge from different datasets, and (2) gain consistent performance gain with more pretraining data. In experiments, we compare our model with discriminative and generative models pretrained on the same dataset. The re- sults confirm that our framework effectively ex- ploits existing annotated data and outperforms baselines in both zero (9.4% average improve- ment) and few shot settings (3.5% average im- provement). Our code is available at https: //github.com/psunlpgroup/ConEntail. 1 Introduction It has been a long-standing effort to solve various text classification tasks by training one universal model (Kumar et al., 2016). With an ideal univer- sal classification model, we can expect extreme generalization with few or zero annotation in new domains/tasks/datasets. To this end, researchers reformulate heterogeneous task definitions into a unified format of a meta-task in natural language (Yin et al., 2020; Khashabi et al., 2020a). Solving the meta-task is equivalent to solving the isolated tasks, thus the meta-task paves the way of supple- menting unsupervised pretrained Language Models (PLM) with additional supervised pretraining, to further absorb knowledge from heterogeneous la- beled data. The success of universal classification models hinges on how well a strong PLM understands nat- ural language meta-task. The meta-task format depends on two underlying PLM types: (a) dis- criminator uses Encoder PLMs and treats all clas- sification tasks as binary entailment classification problem (Yin et al., 2019, 2020; Xia et al., 2021; Wang et al., 2021). However, they only pretrain models on Natural Language Inference datasets, whose knowledge is not comprehensive comparing all classification tasks (Ma et al., 2021). (b) gen- erator uses Encoder-Decoder PLMs and treats all tasks as text generation problem …

Content truncated to 500 words