Re-embedding Difficult Samples via Mutual Information Constrained Semantically Oversampling for Imbalanced Text Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3148–3161 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3148 Difficult Samples Re-embedding via Mutual Information Constrained Semantically Oversampling Jiachen Tian, Shizhan Chen, Xiaowang Zhang∗, Zhiyong Feng, Deyi Xiong, Shaojuan Wu and Chunliu Dou College of Intelligence and Computing, Tianjin University, Tianjin, China {jiachen6677, shizhan, xiaowangzhang, zyfeng, dyxiong, shaojuanwu, 2019229041}@tju.edu.cn Abstract Difficult samples of the minority class in im- balanced text classification are usually hard to be classified as they are embedded into an overlapping semantic region with the majori- ty class. In this paper, we propose a Mutu- al Information constrained Semantically Over- sampling framework (MISO) that can gener- ate anchor instances to help the backbone net- work determine the re-embedding position of a non-overlapping representation for each dif- ficult sample. MISO consists of (1) a semantic fusion module that learns entangled semantics among difficult and majority samples with an adaptive multi-head attention mechanism, (2) a mutual information loss that forces our mod- el to learn new representations of entangled se- mantics in the non-overlapping region of the minority class, and (3) a coupled adversarial encoder-decoder that fine-tunes disentangled semantic representations to remain their corre- lations with the minority class, and then using these disentangled semantic representations to generate anchor instances for each difficult sample. Experiments on a variety of imbal- anced text classification tasks demonstrate that anchor instances help classifiers achieve signif- icant improvements over strong baselines. 1 Introduction Data imbalance is a long-standing challenge in the text classification tasks such as sentiment analy- sis (Wu et al., 2018), intent detection (Quan et al., 2020) and spam detection (Liu et al., 2017), where the distribution of training data over classes is skewed. For example, the number of minority sam- ples accounts for only 28% of training instances in SMS Spam dataset (Peng et al., 2019) and 14% in Opin-Rank dataset (Ganesan and Zhai, 2012). Data imbalanced issue is more severe in Toutiao dataset (Ouyang et al., 2020) with a minority- majority ratio of 1:122 (hereafter imbalance ratio). ∗Corresponding author: xiaowangzhang@tju.edu.cn Figure 1: Visualization of the data imbalance problem. As the class distribution tends to be extremely im- balanced, texts from the minority class(es) may be easily categorized into the majority class(es) (He and Garcia, 2009; Fernández et al., 2018; Gao et al., 2020; Yang et al., 2020). Datasets Perc. (%) of DS | Non-DS F1-score (%) of DS | Non-DS Opin-Rank 15.3 | 84.7 56.2 | 99.3 SMS Spam 42.1 | 57.9 64.3 | 94.6 Toutiao 36.6 | 63.4 59.0 | 95.6 Yelp.P (1%) 28.2 | 71.8 32.6 | 94.3 Yelp.P (5%) 22.4 | 77.6 46.4 | 93.8 IMDB (1%) 49.3 | 50.7 18.8 | 83.6 IMDB (5%) 30.1 | 69.9 26.9 | 87.3 AG_News (1%) 48.5 | 51.5 17.5 | 75.9 AG_News (5%) 33.2 | 66.8 27.7 | 83.8 Table 1: Statistics with respect to imbalanced datasets, and the classification performance of XLNet about dif- ficult and non-difficult samples. DS: difficult samples. Non-DS: non-difficult …

Content truncated to 500 words