Entailment as Robust Self-Learner

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13803–13817 July 9-14, 2023 ©2023 Association for Computational Linguistics Entailment as Robust Self-Learner Jiaxin Ge1∗ and Hongyin Luo2∗ and Yoon Kim2 and James Glass2 1 Peking University, Beijing, China 2 MIT Computer Science and Artificial Intelligence Lab, Cambridge MA, US aomaru@stu.pku.edu.cn, {hyluo, yoonkim, glass}@mit.edu Abstract Entailment has been recognized as an impor- tant metric for evaluating natural language un- derstanding (NLU) models, and recent studies have found that entailment pretraining benefits weakly supervised fine-tuning. In this work, we design a prompting strategy that formulates a number of different NLU tasks as contex- tual entailment. This approach improves the zero-shot adaptation of pretrained entailment models. Secondly, we notice that self-training entailment-based models with unlabeled data can significantly improve the adaptation perfor- mance on downstream tasks. To achieve more stable improvement, we propose the Simple Pseudo-Label Editing (SimPLE) algorithm for better pseudo-labeling quality in self-training. We also found that both pretrained entailment- based models and the self-trained models are robust against adversarial evaluation data. Ex- periments on binary and multi-class classifi- cation tasks show that SimPLE leads to more robust self-training results, indicating that the self-trained entailment models are more effi- cient and trustworthy than large language mod- els on language understanding tasks. 1 Introduction Although achieving state-of-the-art performance in different natural language understanding (NLU) tasks (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019; Clark et al., 2020; He et al., 2020; Joshi et al., 2020), large-scale pretrained language mod- els still highly depend on human-annotated, task- specific training corpora for fine-tuning because the self-supervised pretraining objective does not incorporate explicit task-related knowledge. As a result, state-of-the-art language models are still challenged by the lack of adequate fine-tuning data and difficult evaluation examples crafted by adver- ∗ Equal contribution. Correspondence to Hongyin Luo at hyluo@mit.edu. Code and processed data are available at https://github.com/luohongyin/EntST. sarial attacks or model-in-loop adversarial data an- notations (Wang et al., 2021a; Bartolo et al., 2020; Zang et al., 2019; Garg and Ramakrishnan, 2020; Li et al., 2020). On the other hand, entailment is recognized as a minimal requirement for NLU (Condoravdi et al., 2003). Recent studies have found that en- tailment learning improves sentence representation (Reimers and Gurevych, 2019a; Gao et al., 2021). However, these models still need fine-tuning with human-annotated training data to handle down- stream NLU tasks. The authors of Wang et al. (2021b) found that entailment-based models are also few-shot learners that outperform recent ef- forts on few-shot NLU. For example, LM-BFF (Gao et al., 2020) proves that entailment learning can significantly improve the data efficiency and adaptation ability of language models. In this work, we further explore the zero-shot and unsupervised adaptation abilities of entailment- based models without any human-labeled training corpora on downstream tasks. We first study the zero-shot and unsupervised adaptation abilities of the entailment-based language models. Inspired by recent progress on prompt tuning, we formulate different NLU tasks as contextual …

Content truncated to 500 words