Regularizing Text Categorization with Clusters of Words

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1827–1837, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Regularizing Text Categorization with Clusters of Words Konstantinos Skianis François Rousseau LIX, École Polytechnique, France kskianis@lix.polytechnique.fr Michalis Vazirgiannis Abstract Regularization is a critical step in supervised learning to not only address overfitting, but also to take into account any prior knowledge we may have on the features and their de- pendence. In this paper, we explore state- of-the-art structured regularizers and we pro- pose novel ones based on clusters of words from LSI topics, word2vec embeddings and graph-of-words document representation. We show that our proposed regularizers are faster than the state-of-the-art ones and still improve text classification accuracy. Code and data are available online1. 1 Introduction Harnessing the full potential in text data has always been a key task for the NLP and ML communities. The properties hidden under the inherent high di- mensionality of text are of major importance in tasks such as text categorization and opinion mining. Although simple models like bag-of-words man- age to perform well, the problem of overfitting still remains. Regularization as proven in Chen and Rosenfeld (2000) is of paramount importance in Natural Language Processing and more specifically language modeling, structured prediction, and clas- sification. In this paper we build upon the work of Yogatama and Smith (2014b) who introduce prior knowledge of data as a regularization term. One of the most popular structured regularizers, the group lasso (Yuan and Lin, 2006), was proposed to avoid large L2 norms for groups of weights. 1https://goo.gl/mKqvro In this paper, we propose novel linguistic struc- tured regularizers that capitalize on the clusters learned from texts using the word2vec and graph-of- words document representation, which can be seen as group lasso variants. The extensive experiments we conducted demonstrate these regularizers can boost standard bag-of-words models on most cases tested in the task of text categorization, by imposing additional unused information as bias. 2 Background & Notation We place ourselves in the scenario where we con- sider a prediction problem, in our case text catego- rization, as a loss minimization problem, i. e. we define a loss function L(x, θ, y) that quantifies the loss between the prediction hθ,b(x) of a classifier parametrized by a vector of feature weights θ and a bias b, and the true class label y ∈ Y associated with the example x ∈ X. Given a training set of N data points {(xi, yi)}i=1...N, we want to find the optimal set of feature weights θ∗ such that: θ∗ = argmin θ N � i=1 L(xi, θ, yi) � �� � empirical risk (1) In the case of logistic regression with binary predic- tions (Y = {−1, +1}), hθ,b(x) = θ⊤x + b and L(x, θ, y) = e−yhθ,b(x) (log loss). 2.1 Regularization Only minimizing the empirical risk can lead to over- fitting, that is, the model no longer learns the un- derlying pattern we are trying to capture but fits …

Content truncated to 500 words