Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11923–11936 July 9-14, 2023 ©2023 Association for Computational Linguistics Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge Vasudha Varadarajan♣∗, Swanie Juhng♣∗, Syeda Mahwish♣, Xiaoran Liu♠ Jonah Luby, Christian C. Luhmann♠ and H. Andrew Schwartz♣ ♣Department of Computer Science, ♠Department of Psychology Stony Brook University {vvaradarajan,sjuhng,smahwish,has}@cs.stonybrook.edu {christian.luhmann,xiaoran.liu}@stonybrook.edu, jonahluby@gmail.com Abstract While transformer-based systems have enabled greater accuracies with fewer training exam- ples, data acquisition obstacles still persist for rare-class tasks – when the class label is very infrequent (e.g., < 5% of samples). Active learning has in general been proposed to al- leviate such challenges, but choice of selec- tion strategy, the criteria by which rare-class examples are chosen, has not been system- atically evaluated. Further, transformers en- able iterative transfer-learning approaches. We propose and investigate transfer- and active learning solutions to the rare class problem of dissonance detection through utilizing models trained on closely related tasks and the eval- uation of acquisition strategies, including a proposed probability-of-rare-class (PRC) ap- proach. We perform these experiments for a specific rare class problem: collecting language samples of cognitive dissonance from social media. We find that PRC is a simple and effec- tive strategy to guide annotations and ultimately improve model accuracy, and while transfer- learning in a specific order can improve the cold-start performance of the learner but does not benefit iterations of active learning. 1 Introduction Cognitive dissonance occurs during everyday think- ing when one experiences two or more beliefs that are inconsistent in some way (Harmon-Jones and Harmon-Jones, 2007). Often expressed in lan- guage, dissonance plays a role in many aspects of life, for example affecting health-related behav- ior such as smoking (Chapman et al., 1993) and contributing to the development of (and exit from) extremism (Dalgaard-Nielsen, 2013). However, while the phenomenon is common enough to occur on a daily basis, dissonance is still relatively rare among the myriad of other relationships between * co-lead authors Humans annotate Acquisition strategy: which are “best” to label? Add new examples Model Retrain/Update Active Learning Iteration new examples Cumulative (CM) Iterative (IT) new old data old model fine-tune new old data old model train new model new model Initial model: Transfer Learning START increase chance of rare class? Rare class annotation ~ “needle in a haystack” Easier to annotate Difficult to annotate Figure 1: Demonstration of the active learning (AL) loop in general. Our paper examines the three high- lighted steps: (i) Bootstrapping with TL model, (ii) Acquisition strategy, and (iii) Model update. beliefs that occur across random selections of lin- guistic expressions and thus makes the automatic detection of it a rare-class problem. Despite recent advances in modeling sequences of words, rare-class tasks – when the class label is very infrequent (e.g., < 5% of samples) – re- main challenging due to the low rate of positive examples. Not only are more random examples necessary to reach a substantial amount of the rare class …

Content truncated to 500 words