Sibylvariant Transformations for Robust Text Classification

Findings of the Association for Computational Linguistics: ACL 2022, pages 1771 - 1788 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Sibylvariant Transformations for Robust Text Classification Fabrice Harel-Canada1, Muhammad Ali Gulzar2, Nanyun Peng1, Miryung Kim1 1Computer Science Department, University of California, Los Angeles 2Computer Science Department, Virginia Tech {fabricehc, violetpeng, miryung}@cs.ucla.edu, gulzar@cs.vt.edu Abstract The vast majority of text transformation tech- niques in NLP are inherently limited in their ability to expand input space coverage due to an implicit constraint to preserve the origi- nal class label. In this work, we propose the notion of sibylvariance (SIB) to describe the broader set of transforms that relax the label- preserving constraint, knowably vary the ex- pected class, and lead to significantly more diverse input distributions. We offer a uni- fied framework to organize all data transforma- tions, including two types of SIB: (1) Trans- mutations convert one discrete kind into an- other, (2) Mixture Mutations blend two or more classes together. To explore the role of sibylvariance within NLP, we implemented 41 text transformations, including several novel techniques like Concept2Sentence and SentMix. Sibylvariance also enables a unique form of adaptive training that gener- ates new input mixtures for the most confused class pairs, challenging the learner to differen- tiate with greater nuance. Our experiments on six benchmark datasets strongly support the ef- ficacy of sibylvariance for generalization per- formance, defect detection, and adversarial ro- bustness. 1 Introduction Automatically generating new data is a critical com- ponent within modern machine learning pipelines. During training, data augmentation can expose models to a larger portion of potential input space, consistently leading to better generalization and performance (Simard et al., 1998; Krizhevsky et al., 2012; Perez and Wang, 2017). After training, cre- ating effective test instances from existing data can expose specific model failure modes and provide actionable corrective feedback (Zhang et al., 2019; Ribeiro et al., 2020). While many techniques can artificially expand labeled training sets or test suites, nearly all of them are class-preserving. That is to say, the model out- puts are invariant (INV) with respect to the transfor- mations. This cautious constraint ensures the new data does not lie in an out-of-distribution null class which might impede the learning objective. How- ever, it also requires more conservative transforms that inherently limit the degree of diversification. In this work, we propose and extensively in- vestigate the potential of sibylvariant (SIB) trans- formations that knowably vary the expected class. From the Greek sibyls, or oracles, the term parallels the oracle construction problem in software test- ing (Barr et al., 2015). In a nutshell, sibylvariants either fully transmute a datum from one class ci to another cj, or mix data from multiple classes together to derive a new input with a soft label that reflects the mixed membership. In this way, SIB can more strongly perturb and diversify the under- lying distribution. Moreover, SIB makes possible a new type of adaptive training by synthesizing data from frequently confused class pairs, challenging the model to …

Content truncated to 500 words