CAISA at SemEval-2023 Task 8: Counterfactual Data Augmentation for Mitigating Class Imbalance in Causal Claim Identification

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2118–2123 July 13-14, 2023 ©2023 Association for Computational Linguistics CAISA at SemEval-2023 Task 8: Counterfactual Data Augmentation for Mitigating Class Imbalance in Causal Claim Identification Akbar Karimi1,2 and Lucie Flek1,2 1Conversational AI and Social Analytics (CAISA) Lab, University of Bonn, Germany 2Conversational AI and Social Analytics (CAISA) Lab, University of Marburg, Germany {akkarimi, lucie.flek}@bit.uni-bonn.de https://caisa-lab.github.io Abstract The class imbalance problem can cause ma- chine learning models to produce an undesir- able performance on the minority class as well as the whole dataset. Using data augmentation techniques to increase the number of samples is one way to tackle this problem. We intro- duce a novel counterfactual data augmentation by verb replacement for the identification of medical claims. In addition, we investigate the impact of this method and compare it with 3 other data augmentation techniques, showing that the proposed method can result in a sig- nificant (relative) improvement in the minority class. 1 Introduction Automatic identification of medical claims (Khetan et al., 2023; Wadhwa et al., 2023) is a task with various real-life applications in industries such as healthcare (Herland et al., 2017) and insurance (Wang and Xu, 2018) as well as content moder- ation (Schlicht et al., 2023). However, it can be a difficult task due to the lack of data for all or some categories. One solution for such an issue is increasing the number of data points in each cate- gory, especially the one that has significantly fewer samples. We can do this using data augmentation techniques (Temraz and Keane, 2022), which mod- ify certain characteristics of an input sequence (or its representation in the embedding space) in order to create different versions of it. One example is en- tity replacement (Zeng et al., 2020), where entities in one sequence can be swapped with equivalent ones from another sequence. The advantage of this type of augmentation is that it provides more real context to the target entities. Given that the task at hand is claim detection, we hypothesize that the verb in a sentence can be determinant in its category. Therefore, we address the problem of class imbalance using a novel data augmentation technique where we replace a verb in a sentence with other verbs from the training data. Our experiments show that verb replacement can improve the performance of a model on the target category. In addition, for more comparison, we experiment with several other data augmentation techniques, namely noise insertion (Karimi et al., 2021b), entity replacement (Zeng et al., 2020), aug- mentation with YouChat1, and augmentation in the embedding space (Karimi et al., 2021a). 2 Background Class Imbalance Problem. This problem fre- quently comes up in many domains and applica- tions. As a result, it has been tackled by a variety of methods such as oversampling (Ling and Li, 1998) and undersampling (He and Garcia, 2009). The former method randomly selects some of the sam- ples in the minority class and uses them multiple …

Content truncated to 500 words