Variational Pretraining for Semi-supervised Text Classification

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5880–5894 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5880 Variational Pretraining for Semi-supervised Text Classification Suchin Gururangan1 Tam Dang2 Dallas Card3 Noah A. Smith1,2 1Allen Institute for Artificial Intelligence, Seattle, WA, USA 2Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA 3Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA suching@allenai.org {dangt7,nasmith}@cs.washington.edu dcard@cmu.edu Abstract We introduce VAMPIRE,1 a lightweight pre- training framework for effective text classi- fication when data and computing resources are limited. We pretrain a unigram docu- ment model as a variational autoencoder on in-domain, unlabeled data and use its inter- nal states as features in a downstream classi- fier. Empirically, we show the relative strength of VAMPIRE against computationally expen- sive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in- domain data is crucial to achieving decent per- formance from contextual embeddings when working with limited supervision. We accom- pany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks. 1 Introduction An effective approach to semi-supervised learning has long been a goal for the NLP community, as unlabeled data tends to be plentiful compared to labeled data. Early work emphasized using unla- beled data drawn from the same distribution as the labeled data (Nigam et al., 2000), but larger and more reliable gains have been obtained by using contextual embeddings trained with a language modeling (LM) objective on massive amounts of text from domains such as Wikipedia or news (Pe- ters et al., 2018a; Devlin et al., 2019; Radford et al., 2018; Howard and Ruder, 2018). The latter approaches play to the strengths of high-resource settings (e.g., access to web-scale corpora and powerful machines), but their computational and data requirements can make them less useful in resource-limited environments. In this paper, we instead focus on the low-resource setting (§2.1), 1VAriational Methods for Pretraining In Resource-limited Environments and develop a lightweight approach to pretraining for semi-supervised text classification. Our model, which we call VAMPIRE, com- bines a variational autoencoder (VAE) approach to document modeling (Kingma and Welling, 2013; Miao et al., 2016; Srivastava and Sutton, 2017) with insights from LM pretraining (Peters et al., 2018a). By operating on a bag-of-words represen- tation, we avoid the time complexity and difficulty of training a sequence-to-sequence VAE (Bowman et al., 2016; Xu et al., 2017; Yang et al., 2017) while retaining the freedom to use a multi-layer encoder that can learn useful representations for downstream tasks. Because VAMPIRE ignores se- quential information, it leads to models that are much cheaper to train, and offers strong perfor- mance when the amount of labeled data is small. Finally, because VAMPIRE is a descendant of topic models, we are able to explore model selection by topic coherence, rather than validation-set per- plexity, which results in better downstream classi- fication performance (§6.1). In order to evaluate the …

Content truncated to 500 words