Prototype-Guided Pseudo Labeling for Semi-Supervised Text Classification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 16369–16382 July 9-14, 2023 ©2023 Association for Computational Linguistics Prototype-Guided Pseudo Labeling for Semi-Supervised Text Classification Weiyi Yang1, Richong Zhang1,2∗, Junfan Chen1, Lihong Wang3, Jaein Kim1 1SKLSDE, School of Computer Science and Engineering, Beihang University, Beijing, China 2Zhongguancun Laboratory, Beijing, China 3CNCERT Abstract The semi-supervised text classification (SSTC) task aims at training text classification mod- els with a few labeled data and massive unla- beled data. Recent works achieve this task by pseudo-labeling methods that assign pseudo- labels to unlabeled data as additional supervi- sion. However, these models may suffer from incorrect pseudo-labels caused by underfitting of decision boundaries and generating biased pseudo-labels on imbalanced data. We propose a prototype-guided semi-supervised model to address the above problems, which integrates a prototype-anchored contrasting strategy and a prototype-guided pseudo-labeling strategy. Par- ticularly, the prototype-anchored constrasting constructs prototypes to cluster text represen- tations with the same class, forcing them to be high-density distributed, thus alleviating the underfitting of decision boundaries. And the prototype-guided pseudo-labeling selects re- liable pseudo-labeled data around prototypes based on data distribution, thus alleviating the bias from imbalanced data. Empirical results on 4 commonly-used datasets demonstrate that our model is effective and outperforms state-of- the-art methods. 1 Introduction Traditional text classification has achieved pro- found success by leveraging numerous labeled data. However, acquiring large-scale labeled data is ex- pensive in many real-world scenarios, making train- ing effective classifiers using such approaches diffi- cult. Therefore, semi-supervised text classification (SSTC), aiming at text classification requiring a few labeled data and massive unlabeled data, has become a new appealing technique. As a promising semi-supervised paradigm, the pseudo-labeling method (Mukherjee and Awadal- lah, 2020; Xie et al., 2020; che; Lee et al., 2021; Li ∗Corresponding author: zhangrc@act.buaa.edu.cn (a) (b) c2c2 c2c2 c1c1 c1c1 c3c3 c3c3 Figure 1: Illustration of two cases of pseduo-labeling problems in existing SSTC models: (a) underfitting near decision boundaries; (b) bias from imbalanced data. ⋆ denotes a prototype consisting of labeled data. Different symbols indicate different categories of labeled (bor- dered) and unlabeled data (unbordered). The areas with different colors indicate different categories. et al., 2022), which assigns pseudo-labels to unla- beled data instances as additional supervision, has drawn attention. They follow the well-known low- density separation assumption (Chapelle and Zien, 2005) to generate pseudo-labels for the unlabeled data scattered in the high-density feature space, helping to identify the boundaries between classes. Trained with the mixture of labeled and pseudo- labeled data, the model perceives more abundant supervision signals and reduces empirical errors. Existing models attempt to assign soft pseudo- labels for unlabeled texts (Xie et al., 2020) or keep pseudo-labels with lower uncertainty (Mukherjee and Awadallah, 2020) to alleviate the influence brought by the incorrect pseudo-labeled data. In practice, the underfitting of decision bound- aries is usually detrimental to the model learn- ing (shown in Figure 1(a)). More careful consid- erations are expected in properly utilizing these pseudo-labeled data near decision boundaries. …

Content truncated to 500 words