MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147–2157 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2147 MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification Jiaao Chen Georgia Tech jchen896@gatech.edu Zichao Yang CMU zichaoy@cs.cmu.edu Diyi Yang Georgia Tech dyang888@gatech.edu Abstract This paper presents MixText, a semi- supervised learning method for text classifi- cation, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data. By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of- the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https: //github.com/GT-SALT/MixText. 1 Introduction In the era of deep learning, research has achieved extremely good performance in most supervised learning settings (LeCun et al., 2015; Yang et al., 2016). However, when there is only limited labeled data, supervised deep learning models often suffer from over-fitting (Xie et al., 2019). This strong dependence on labeled data largely prevents neural network models from being applied to new settings or real-world situations due to the need of large amount of time, money, and expertise to obtain enough labeled data. As a result, semi-supervised learning has received much attention to utilize both labeled and unlabeled data for different learning tasks, as unlabeled data is always much easier and cheaper to collect (Chawla and Karakoulas, 2011). This work takes a closer look at semi-supervised text classification, one of the most fundamental tasks in language technology communities. Prior research on semi-supervised text classification can Figure 1: TMix takes in two text samples x and x′ with labels y and y′, mixes their hidden states h and h′ at layer m with weight λ into ˜h, and then continues for- ward passing to predict the mixed labels ˜y. be categorized into several classes: (1) utilizing variational auto encoders (VAEs) to reconstruct the sentences and predicting sentence labels with la- tent variables learned from reconstruction such as (Chen et al., 2018; Yang et al., 2017; Gururangan et al., 2019); (2) encouraging models to output confident predictions on unlabeled data for self- training like (Lee, 2013; Grandvalet and Bengio, 2004; Meng et al., 2018); (3) performing consis- tency training after adding adversarial noise (Miy- ato et al., 2019, 2017) or data augmentations (Xie et al., 2019); (4) large scale pretraining with unla- beld data, then finetuning with labeled data (Devlin et al., 2019). Despite the huge success of those models, most prior work utilized labeled and unla- beled data separately in a way that no supervision can transit from labeled to unlabeled data or from unlabeled to labeled data. As a result, most semi- supervised models can easily still overfit on the …

Content truncated to 500 words