Semi-Supervised Text Classification with Balanced Deep Representation Distributions

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5044–5053 August 1–6, 2021. ©2021 Association for Computational Linguistics 5044 Semi-Supervised Text Classification with Balanced Deep Representation Distributions Changchun Li, Ximing Li∗† and Jihong Ouyang† College of Computer Science and Technology, Jilin University, China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, China changchunli93@gmail.com, liximing86@gmail.com, ouyj@jlu.edu.cn Abstract Semi-Supervised Text Classification (SSTC) mainly works under the spirit of self-training. They initialize the deep classifier by training over labeled texts; and then alternatively pre- dict unlabeled texts as their pseudo-labels and train the deep classifier over the mixture of labeled and pseudo-labeled texts. Naturally, their performance is largely affected by the accuracy of pseudo-labels for unlabeled texts. Unfortunately, they often suffer from low ac- curacy because of the margin bias problem caused by the large difference between repre- sentation distributions of labels in SSTC. To alleviate this problem, we apply the angular margin loss, and perform Gaussian linear trans- formation to achieve balanced label angle vari- ances, i.e., the variance of label angles of texts within the same label. More accuracy of pre- dicted pseudo-labels can be achieved by con- straining all label angle variances balanced, where they are estimated over both labeled and pseudo-labeled texts during self-training loops. With this insight, we propose a novel SSTC method, namely Semi-Supervised Text Clas- sification with Balanced Deep representation Distributions (S2TC-BDD). To evaluate S2TC- BDD, we compare it against the state-of-the- art SSTC methods. Empirical results demon- strate the effectiveness of S2TC-BDD, espe- cially when the labeled texts are scarce. 1 Introduction Semi-Supervised Learning (SSL) refers to the paradigm of learning with labeled as well as un- labeled data to perform certain applications (van Engelen and Hoos, 2020). Especially, developing effective SSL models for classifying text data has long been a goal for the studies of natural language processing, because labeled texts are difficult to col- lect in many real-world scenarios. Formally, this ∗ Contributing equally with the first author. † Corresponding author. Figure 1: The average difference of label angle vari- ances (Avg.DLAV) computed in semi-supervised and supervised manners across AG News, respectively. research topic is termed as Semi-Supervised Text Classification (SSTC), which nowadays draws much attention from the community (Clark et al., 2018; Gururangan et al., 2019; Chen et al., 2020). To our knowledge, the most recent SSTC meth- ods mainly borrow ideas from the successful pat- terns of supervised deep learning, such as pre- training and fine-tuning (Dai and Le, 2015; Howard and Ruder, 2018; Peters et al., 2018; Gururangan et al., 2019; Devlin et al., 2019). Generally, those methods perform deep representation learning on unlabeled texts followed by supervised learning on labeled texts. However, a drawback is that they separately learn from the labeled and unlabeled texts, where, specifically, the deep representations are trained without using the labeling information, resulting in potentially less discriminative represen- tations as well as worse performance. To avoid …

Content truncated to 500 words