Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11778–11801 July 9-14, 2023 ©2023 Association for Computational Linguistics Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Models Albert Xu Xiang Ren Robin Jia University of Southern California {albertxu,xiangren,robinjia}@usc.edu Abstract In many task settings, text classification mod- els are likely to encounter examples from novel classes on which they cannot predict correctly. Selective prediction, in which models abstain on low-confidence examples, provides a pos- sible solution, but existing models are often overly confident on unseen classes. To remedy this overconfidence, we introduce Contrastive Novelty-Augmented Learning (CoNAL), a two- step method that generates OOD examples rep- resentative of novel classes, then trains to de- crease confidence on them. First, we generate OOD examples by prompting a large language model twice: we prompt it to enumerate rel- evant novel classes, then generate examples from each novel class matching the task for- mat. Second, we train a classifier with a novel contrastive objective that encourages lower con- fidence on generated OOD examples than train- ing examples. When trained with CoNAL, clas- sifiers improve in their ability to detect and ab- stain on novel class examples over prior meth- ods by an average of 2.3% in terms of accuracy under the accuracy-coverage curve (AUAC) and 5.5% AUROC across 4 NLP datasets, with no cost to in-distribution accuracy.1 1 Introduction Recent progress in NLP has led to text clas- sification models that are accurate not only in-distribution (ID), but also on some out-of- distribution (OOD) data (Arora et al., 2021). Nonetheless, some categories of real-world dis- tribution shift still pose serious challenges. For instance, in open-set label shift (Garg et al., 2022), the test data includes examples from novel classes not present in the training data, making it impos- sible for a standard classifier to predict correctly (Scheirer et al., 2013). Moreover, novel class exam- ples can be difficult to detect with conventional OOD detection methods, as they typically bear 1Code is available at github.com/albertkx/CoNAL. a strong surface resemblance to training exam- ples (¸Tifrea et al., 2021). In this paper, we frame open-set label shift as a selective prediction prob- lem (El-Yaniv and Wiener, 2010; Geifman and El- Yaniv, 2017) that we call open-set selective clas- sification (OSSC). OSSC requires text classifiers to predict correctly on closed-set examples while abstaining on novel class examples. To perform well on OSSC, a classifier must have lower confidence on novel class examples than closed-set examples by learning features which differentiate novel classes from closed-set classes (Perera et al., 2020). In order to supervise this representation learning, it is useful to identify what examples from novel classes might look like. Prior work has explored automatically generating OOD images by adding random perturbations to ID examples (Setlur et al., 2022). Text inputs, how- ever, are composed of discrete tokens, and modi- fying even a single token can unpredictably alter the meaning of a sentence. We seek an automatic generation method that addresses these …

Content truncated to 500 words