Linguistic Structured Sparsity in Text Categorization

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 786–796, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Linguistic Structured Sparsity in Text Categorization Dani Yogatama Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA dyogatama@cs.cmu.edu Noah A. Smith Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA nasmith@cs.cmu.edu Abstract We introduce three linguistically moti- vated structured regularizers based on parse trees, topics, and hierarchical word clusters for text categorization. These regularizers impose linguistic bias in fea- ture weights, enabling us to incorporate prior knowledge into conventional bag- of-words models. We show that our structured regularizers consistently im- prove classification accuracies compared to standard regularizers that penalize fea- tures in isolation (such as lasso, ridge, and elastic net regularizers) on a range of datasets for various text prediction prob- lems: topic classification, sentiment anal- ysis, and forecasting. 1 Introduction What is the best way to exploit linguistic infor- mation in statistical text processing models? For tasks like text classification, sentiment analysis, and text-driven forecasting, this is an open ques- tion, as cheap “bag-of-words” models often per- form well. Much recent work in NLP has fo- cused on linguistic feature engineering (Joshi et al., 2010) or representation learning (Glorot et al., 2011; Socher et al., 2013). In this paper, we propose a radical alternative. We embrace the conventional bag-of-words repre- sentation of text, instead bringing linguistic bias to bear on regularization. Since the seminal work of Chen and Rosenfeld (2000), the importance of regularization in discriminative models of text— including language modeling, structured predic- tion, and classification—has been widely recog- nized. The emphasis, however, has largely been on one specific kind of inductive bias: avoiding large weights (i.e., coefficients in a linear model). Recently, structured (or composite) regulariza- tion has been introduced; simply put, it reasons about different weights jointly. The most widely explored variant, group lasso (Yuan and Lin, 2006) seeks to avoid large ℓ2 norms for groups of weights. Group lasso has been shown useful in a range of applications, including computational biology (Kim and Xing, 2008), signal processing (Lv et al., 2011), and NLP (Eisenstein et al., 2011; Martins et al., 2011; Nelakanti et al., 2013). For text categorization problems, Yogatama and Smith (2014) proposed groups based on sentences, an idea generalized here to take advantage of richer linguistic information. In this paper, we show how linguistic informa- tion of various kinds—parse trees, thematic topics, and hierarchical word clusterings—can be used to construct group lasso variants that impose linguis- tic bias without introducing any new features. Our experiments demonstrate that structured regulariz- ers can squeeze higher performance out of conven- tional bag-of-words models on seven out of eight of text categorization tasks tested, in six cases with more compact models than the best-performing unstructured-regularized model. 2 Notation We represent each document as a feature vector x ∈ RV , where V is the vocabulary size. xv is the frequency of the …

Content truncated to 500 words