EICO: Improving Few-Shot Text Classification via Explicit and Implicit Consistency Regularization

Findings of the Association for Computational Linguistics: ACL 2022, pages 3582 - 3587 May 22-27, 2022 c⃝2022 Association for Computational Linguistics EICO: Improving Few-Shot Text Classification via Explicit and Implicit Consistency Regularization Lei Zhao Cheng Yao Alibaba Group {sashi.zl,cheng.yaoc}@alibaba-inc.com Abstract While the prompt-based fine-tuning methods had advanced few-shot natural language un- derstanding tasks, self-training methods are also being explored. This work revisits the consistency regularization in self-training and presents explicit and implicit consistency reg- ularization enhanced language model (EICO). By employing both explicit and implicit con- sistency regularization, EICO advances the per- formance of prompt-based few-shot text clas- sification. For implicit consistency regulariza- tion, we generate pseudo-label from the weakly- augmented view and predict pseudo-label from the strongly-augmented view. For explicit con- sistency regularization, we minimize the dif- ference between the prediction of the augmen- tation view and the prediction of the original view. We conducted extensive experiments on six text classification datasets and found that with sixteen labeled examples, EICO achieves competitive performance compared to existing self-training few-shot learning methods. 1 Introduction Recently, (Schick and Schütze, 2021a,b) proposed a cloze-style few-shot learning method, PET. By filling the gap between pre-training and fine-tuning with prompt and verbalizer, PET achieved com- petitive performance with GPT-3 (Brown et al., 2020) with smaller language models like BERT (Devlin et al., 2019), T5 (Lester et al., 2021) and GPT-2 (Radford et al.; Liu et al., 2021). How- ever, the lack of labeled data still limits the perfor- mance of few-shot learning. Although acquiring la- beled data is costly, unlabeled data is relatively easy to obtain. Leveraging unlabeled data to improve the performance of the few-shot language model through semi-supervised learning is a promising way. In this paper, we focus on advancing the performance of the few-shot language model via semi-supervised learning. Semi-supervised learning is a method that can leverage both labeled and unlabeled data. A com- mon yet effective semi-supervised method is self- learning, which uses a trained teacher model to gen- erate pseudo-labels for unlabeled examples, and then trains a student model on both labeled and pseudo-labeled examples to utilize the domain- relevant information contained in the unlabeled data. FixMatch (Sohn et al., 2020) and its nat- ural language processing adaption SFLM (Chen et al., 2021) are robust self-training implementa- tions, which generate pseudo-label on the weakly- augmented view of an example and predict pseudo- label by the strongly-augmented view. How- ever, the implicit consistency regularization intro- duced by the above training scheme may be sub- optimal since class distribution information is lost in pseudo-label generation. To address the above problem, we propose ex- plicit and implicit consistency regularization en- hanced language model (EICO). Beside implicit consistency regularization, EICO utilizes explicit consistency regularization by minimizing the differ- ence between the prediction of the augmented view and the prediction of the original view. To validate the effectiveness and robustness of EICO, we con- duct extensive experiments on six natural language understanding tasks. The result of our experiments confirms that EICO can be leveraged to …

Content truncated to 500 words