Learning a Deep Hybrid Model for Semi-Supervised Text Classification

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 471–481, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Learning a Deep Hybrid Model for Semi-Supervised Text Classification Alexander G. Ororbia II, C. Lee Giles, David Reitter College of Information Sciences and Technology The Pennsylvania State University, University Park, PA {ago109, giles, reitter}@psu.edu Abstract We present a novel fine-tuning algorithm in a deep hybrid architecture for semi- supervised text classification. During each increment of the online learning pro- cess, the fine-tuning algorithm serves as a top-down mechanism for pseudo-jointly modifying model parameters following a bottom-up generative learning pass. The resulting model, trained under what we call the Bottom-Up-Top-Down learning al- gorithm, is shown to outperform a vari- ety of competitive models and baselines trained across a wide range of splits be- tween supervised and unsupervised train- ing data. 1 Introduction Recent breakthroughs in learning expressive neu- ral architectures have addressed challenging prob- lems in domains such as computer vision, speech recognition, and natural language processing. This success is owed to the representational power af- forded by deeper architectures supported by long- standing theoretical arguments (Hastad, 1987). These architectures efficiently model complex, highly varying functions via multiple layers of non-linearities, which would otherwise require very “wide” shallow models that need large quan- tities of samples (Bengio, 2012). However, many of these deeper models have relied on mini-batch training on large-scale, labeled data-sets, either us- ing unsupervised pre-training (Bengio et al., 2007) or improved architectural components (such as ac- tivation functions) (Schmidhuber, 2015). In an online learning problem, samples are pre- sented to the learning architecture at a given rate (usually with one-time access to these data points), and, as in the case of a web crawling agent, most of these are unlabeled. Given this, batch training and supervised learning frameworks are no longer applicable. While incremental approaches such as co-training have been employed to help these models learn in a more update-able fashion (Blum and Mitchell, 1998; Gollapalli et al., 2013), neural architectures can naturally be trained in an online manner through the use of stochastic gradient de- scent (SGD). Semi-supervised online learning does not only address practical applications, but it also reflects some challenges of human category acquisition (Tomasello, 2001). Consider the case of a child learning to discriminate between object categories and mapping them to words, given only a small amount of explicitly labeled data (the mother pointing to the object), and a large portion of un- supervised learning, where the child comprehends an adult’s speech or experiences positive feedback for his or her own utterances regardless of their correctness. The original argument in this respect applied to grammar (e.g., Chomsky, 1980; Pullum & Scholz, 2002). While neural networks are not necessarily models of actual cognitive processes, semi-supervised models can show learnability and illustrate possible constraints inherent to the learn- ing process. The contribution of this paper is the develop- ment of the Bottom-Up-Top-Down learning al- gorithm for training a Stacked Boltzmann Ex- perts …

Content truncated to 500 words