Text Classification Using Label Names Only: A Language Model Self-Training Approach

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9006–9017, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9006 Text Classification Using Label Names Only: A Language Model Self-Training Approach Yu Meng1, Yunyi Zhang1, Jiaxin Huang1, Chenyan Xiong2, Heng Ji1, Chao Zhang3, Jiawei Han1 1University of Illinois at Urbana-Champaign, IL, USA 2Microsoft Research, WA, USA 3Georgia Institute of Technology, GA, USA 1{yumeng5, yzhan238, jiaxinh3, hengji, hanj}@illinois.edu 2chenyan.xiong@microsoft.com 3chaozhang@gatech.edu Abstract Current text classification methods typically require a good number of human-labeled doc- uments as training data, which can be costly and difficult to obtain in real applications. Hu- mans can perform classification without see- ing any labeled examples but only based on a small set of words describing the categories to be classified. In this paper, we explore the potential of only using the label name of each class to train classification models on un- labeled data, without using any labeled doc- uments. We use pre-trained neural language models both as general linguistic knowledge sources for category understanding and as rep- resentation learning models for document clas- sification. Our method (1) associates semanti- cally related words with the label names, (2) finds category-indicative words and trains the model to predict their implied categories, and (3) generalizes the model via self-training. We show that our model achieves around 90% ac- curacy on four benchmark datasets including topic and sentiment classification without us- ing any labeled documents but learning from unlabeled data supervised by at most 3 words (1 in most cases) per class as the label name1. 1 Introduction Text classification is a classic and fundamental task in Natural Language Processing (NLP) with a wide spectrum of applications such as question answer- ing (Rajpurkar et al., 2016), spam detection (Jin- dal and Liu, 2007) and sentiment analysis (Pang et al., 2002). Building an automatic text classifica- tion model has been viewed as a task of training machine learning models from human-labeled doc- uments. Indeed, many deep learning-based clas- sifiers including CNNs (Kim, 2014; Zhang et al., 2015) and RNNs (Tang et al., 2015a; Yang et al., 1Source code can be found at https://github.com/ yumeng5/LOTClass. 2016) have been developed and achieved great success when trained on large-scale labeled doc- uments (usually over tens of thousands), thanks to their strong representation learning power that effectively captures the high-order, long-range se- mantic dependency in text sequences for accurate classification. Recently, increasing attention has been paid to semi-supervised text classification which requires a much smaller amount of labeled data. The success of semi-supervised methods stems from the usage of abundant unlabeled data: Unlabeled documents provide natural regularization for constraining the model predictions to be invariant to small changes in input (Chen et al., 2020; Miyato et al., 2017; Xie et al., 2019), thus improving the generalization ability of the model. Despite mitigating the annota- tion burden, semi-supervised methods still require manual efforts from domain experts, which might be difficult or expensive to obtain especially when the number of classes is large. Contrary …

Content truncated to 500 words