Neural Self-Training through Spaced Repetition

Proceedings of NAACL-HLT 2019, pages 21–31 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 21 Neural Self-Training through Spaced Repetition Hadi Amiri Department of Biomedical Informatics, Harvard 10 Shattuck Street, Boston, MA 02115 hadi@hms.harvard.edu Abstract Self-training is a semi-supervised learning ap- proach for utilizing unlabeled data to create better learners. The efficacy of self-training al- gorithms depends on their data sampling tech- niques. The majority of current sampling tech- niques are based on predetermined policies which may not effectively explore the data space or improve model generalizability. In this work, we tackle the above challenges by introducing a new data sampling technique based on spaced repetition that dynamically samples informative and diverse unlabeled in- stances with respect to individual learner and instance characteristics. The proposed model is specifically effective in the context of neu- ral models which can suffer from overfitting and high-variance gradients when trained with small amount of labeled data. Our model outperforms current semi-supervised learning approaches developed for neural networks on publicly-available datasets. 1 Introduction It is often expensive or time-consuming to ob- tain labeled data for Natural Language Processing tasks. In addition, manually-labeled datasets may not contain enough samples for downstream data analysis or novelty detection (Wang and Hebert, 2016). To tackle these issues, semi-supervised learning (Zhu, 2006; Chapelle et al., 2009) has be- come an important topic when one has access to small amount of labeled data and large amount of unlabeled data. Self-training is a type of semi-supervised learn- ing in which a downstream learner (e.g. a clas- sifier) is first trained with labeled data, then the trained model is applied to unlabeled data to gen- erate more labeled instances. A select sample of these instances together with their pseudo (pre- dicted) labels are added to the labeled data and the learner is re-trained using the new labeled dataset. This process repeats until there is no more unla- beled data left or no improvement is observed in model performance on validation data (Zhu, 2006; Zhu and Goldberg, 2009). Conventional self-training methods often rely on prediction confidence of their learners to sam- ple unlabeled data. Typically the most confident unlabeled instances are selected (HEARST, 1991; Yarowsky, 1995; Riloff and Jones, 1999; Zhou et al., 2012). This strategy often causes only those unlabeled instances that match well with the current model being selected during self-training, therefore, the model may fail to best generalize to complete sample space (Zhang and Rudnicky, 2006; Wu et al., 2018). Ideally, a self-training al- gorithm should explore the space thoroughly for better generalization and higher performance. Re- cently Wu et al. (2018) developed an effective data sampling technique for “co-training” (Blum and Mitchell, 1998) methods which require two dis- tinct views of data. Although effective, this model can’t be readily applied to some text datasets due to the two distinct view requirement. In the context of neural networks, pretraining is an effective semi-supervised approach in which layers of a network are first pretrained by learning to …

Content truncated to 500 words