Delta-training: Simple Semi-Supervised Text Classification using Pretrained Word Embeddings

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3458–3463, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3458 Delta-training: Simple Semi-Supervised Text Classification using Pretrained Word Embeddings Hwiyeol Jo Seoul National University hwiyeolj@gmail.com Ceyda Cinarel Seoul National University snu.ceyda@gmail.com Abstract We propose a novel and simple method for semi-supervised text classification. The method stems from the hypothesis that a clas- sifier with pretrained word embeddings always outperforms the same classifier with randomly initialized word embeddings, as empirically observed in NLP tasks. Our method first builds two sets of classifiers as a form of model en- semble, and then initializes their word embed- dings differently: one using random, the other using pretrained word embeddings. We focus on different predictions between the two clas- sifiers on unlabeled data while following the self-training framework. We also use early- stopping in meta-epoch to improve the per- formance of our method. Our method, Delta- training, outperforms the self-training and the co-training framework in 4 different text clas- sification datasets, showing robustness against error accumulation. 1 Introduction 1.1 Motivation Classifiers using deep learning algorithms have performed well in various NLP tasks, but the per- formance is not always satisfactory when utilizing small data. It is necessary to collect more data for acquiring better performance. Although collecting unlabeled text data is relatively easy, labeling in and of itself requires a considerable amount of hu- man labor. In order to incorporate unlabeled data into a task, we have to label the data in accordance to class policies of the task, but the labeling pro- cess requires not only human labor but also do- main knowledge on the classes. Semi-supervised learning (Li and Liu, 2003; Zhu, 2006; Chapelle et al., 2009) can be considered a potential solution that utilizes both labeled data and unlabeled data when building a classifier. The simplest form of semi-supervised learning is self- training (Yarowsky, 1995), which first builds a classifier using labeled data, and then label the un- labeled data. After which the most confident label prediction is added to training set and the process is repeated. The unlabeled data can help address data sparsity, but classification errors might be ac- cumulated along the process. We combine self-training with the hypothesis that a classifier with pretrained word embeddings (memb) is always better than a classifier with ran- domly initialized word embeddings (mrand), as empirically observed in various NLP tasks (Turian et al., 2010). Our method follows the self-training framework but rather focuses on the different pre- dictions of two sets of classifiers on unlabeled data. Therefore we can filter out incorrectly pre- dicted data and correctly predicted data by both classifiers, which are less informative to the clas- sifiers. On the other hand, differently predicted data are much more informative since much of the performance gap between the classifiers come from the different predictions. Although the dif- ferently predicted data may introduce some noise like correctly …

Content truncated to 500 words