FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2481–2491 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2481 FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks Chen Liu, Mengchao Zhang, Zhibin Fu, Pan Hou, Yu Li∗ Beijing Value Simplex Technology Co. Ltd liuchen1350@gmail.com liyu@entropyreduce.com Abstract In natural language processing (NLP), state- of-the-art (SOTA) semi-supervised learning (SSL) frameworks have shown great perfor- mance on deep pre-trained language models such as BERT, and are expected to signifi- cantly reduce the demand for manual label- ing. However, our empirical studies indi- cate that these frameworks are not suitable for lightweight models such as TextCNN, LSTM and etc. In this work, we develop a new SSL framework called FLiText, which stands for Faster and Lighter semi-supervised Text clas- sification. FLiText introduces an inspirer net- work together with the consistency regulariza- tion framework, which leverages a generalized regular constraint on the lightweight models for efficient SSL. As a result, FLiText obtains new SOTA performance for lightweight mod- els across multiple SSL benchmarks on text classification. Compared with existing SOTA SSL methods on TextCNN, FLiText improves the accuracy of lightweight model TextCNN from 51.00% to 90.49% on IMDb, 39.8% to 58.06% on Yelp-5, and from 55.3% to 65.08% on Yahoo. In addition, compared with the fully supervised method on the full dataset, FLi- Text just uses less than 1% of labeled data to improve the accuracy by 6.59%, 3.94%, and 3.22% on the datasets of IMDb, Yelp-5, and Yahoo respectively. 1 Introduction Developments in deep learning technology have great breakthroughs in most natural language pro- cessing (NLP) tasks, such as machine transla- tion, sentiment analysis, and reading comprehen- sion. (Devlin et al., 2019; Yang et al., 2019; Li et al., 2020; Ji et al., 2020; Zhong et al., 2020; Tao et al., 2019; Zhang et al., 2019; Wang et al., 2019; Tian et al., 2020a,b) The success of these advancements is highly dependent on large-scale ∗Corresponding author. and high-quality manual labeled data. However, obtaining vast amounts of high-quality labeled data is expensive. Especially in certain fields, such as fi- nance, medicine, law, and so on, text labeling relies on the in-depth participation of field experts. The rapid development of SSL technology is expected to significantly reduce the demand for labeled data. The core goal of this technology is to use a small number of labeled data and vast amounts of unla- beled data to train a model with good generaliza- tion performance to solve machine-learning prob- lems. (Lee, 2013; Laine and Aila, 2016; Miyato et al., 2018; Tarvainen and Valpola, 2017; Berth- elot et al., 2019b; Xie et al., 2020; Sohn et al., 2020; Berthelot et al., 2019a) Unsupervised data Figure 1: Comparison of the scale of parameters be- tween FLiText and pre-trained language models in re- cent years. augmentation (UDA) (Xie et al., 2020) and Mix- Text (Chen et al., 2020) are SOTA SSL methods for text classification, and have been used to various tasks with notable …

Content truncated to 500 words