Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11386 - 11400 December 7-11, 2022 ©2022 Association for Computational Linguistics Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals Maarten De Raedt♢♣ Fréderic Godin♢ Chris Develder♣ Thomas Demeester♣ ♢ Sinch Chatlayer ♣ Ghent University {maarten.deraedt, chris.develder, thomas.demeester}@ugent.be frederic.godin@sinch.com Abstract For text classification tasks, finetuned language models perform remarkably well. Yet, they tend to rely on spurious patterns in training data, thus limiting their performance on out- of-distribution (OOD) test data. Among recent models aiming to avoid this spurious pattern problem, adding extra counterfactual samples to the training data has proven to be very ef- fective. Yet, counterfactual data generation is costly since it relies on human annotation. Thus, we propose a novel solution that only re- quires annotation of a small fraction (e.g., 1%) of the original training data, and uses automatic generation of extra counterfactuals in an encod- ing vector space. We demonstrate the effective- ness of our approach in sentiment classification, using IMDb data for training and other sets for OOD tests (i.e., Amazon, SemEval and Yelp). We achieve noticeable accuracy improvements by adding only 1% manual counterfactuals: +3% compared to adding +100% in-distribution training samples, +1.3% compared to alternate counterfactual approaches. 1 Introduction and Related Work For a wide range of text classification tasks, finetun- ing large pretrained language models (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020; Lewis et al., 2020) on task-specific data has been proven very effective. Yet, analysis has shown that their predictions tend to rely on spurious patterns (Poliak et al., 2018; Gururangan et al., 2018; Kiritchenko and Mohammad, 2018; McCoy et al., 2019; Niven and Kao, 2019; Zmigrod et al., 2019; Wang and Culotta, 2020), i.e., features that from a human per- spective are not indicative for the classifier’s label. For instance, Kaushik et al. (2019) found the rather neutral words “will”, “my” and “has” to be impor- tant for a positive sentiment classification. Such reliance on spurious patterns were suspected to de- grade performance on out-of-distribution (OOD) test data, distributionally different from training Original source documents Manually created counterfactuals (paired with an original) + + + + + + + + + + + + − − − − Document representation vector space Artifcially created counterfactual representations Fig. 1: We propose to generate counterfactuals in rep- resentation space, learning — from only a few manu- ally created counterfactuals — a mapping function t to transform a document representation ϕ(x) to a coun- terfactual one (having the opposite classification label). Illustration for positively labeled originals only. data (Quiñonero-Candela et al., 2008). Specifically for sentiment classification, this suspicion has been confirmed by Kaushik et al. (2019, 2020); Katakkar et al. (2021). For mitigating the spurious pattern effect, generic methods include regularization of masked language models, which limits over-reliance on a limited set of keywords (Moon et al., 2021). Al- ternatively, to improve robustness in imbalanced data settings, additional training samples can be automatically created (Han …

Content truncated to 500 words