Improved Bayesian Logistic Supervised Topic Models with Data Augmentation

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 187–195, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Improved Bayesian Logistic Supervised Topic Models with Data Augmentation Jun Zhu, Xun Zheng, Bo Zhang Department of Computer Science and Technology TNLIST Lab and State Key Lab of Intelligent Technology and Systems Tsinghua University, Beijing, China {dcszj,dcszb}@tsinghua.edu.cn; vforveri.zheng@gmail.com Abstract Supervised topic models with a logistic likelihood have two issues that potential- ly limit their practical use: 1) response variables are usually over-weighted by document word counts; and 2) existing variational inference methods make strict mean-field assumptions. We address these issues by: 1) introducing a regularization constant to better balance the two parts based on an optimization formulation of Bayesian inference; and 2) developing a simple Gibbs sampling algorithm by intro- ducing auxiliary Polya-Gamma variables and collapsing out Dirichlet variables. Our augment-and-collapse sampling algorithm has analytical forms of each conditional distribution without making any restrict- ing assumptions and can be easily paral- lelized. Empirical results demonstrate sig- nificant improvements on prediction per- formance and time efficiency. 1 Introduction As widely adopted in supervised latent Dirichlet allocation (sLDA) models (Blei and McAuliffe, 2010; Wang et al., 2009), one way to improve the predictive power of LDA is to define a like- lihood model for the widely available document- level response variables, in addition to the likeli- hood model for document words. For example, the logistic likelihood model is commonly used for bi- nary or multinomial responses. By imposing some priors, posterior inference is done with the Bayes’ rule. Though powerful, one issue that could limit the use of existing logistic supervised LDA models is that they treat the document-level response vari- able as one additional word via a normalized like- lihood model. Although some special treatment is carried out on defining the likelihood of the single response variable, it is normally of a much small- er scale than the likelihood of the usually tens or hundreds of words in each document. As noted by (Halpern et al., 2012) and observed in our ex- periments, this model imbalance could result in a weak influence of response variables on the topic representations and thus non-satisfactory predic- tion performance. Another difficulty arises when dealing with categorical response variables is that the commonly used normal priors are no longer conjugate to the logistic likelihood and thus lead to hard inference problems. Existing approaches re- ly on variational approximation techniques which normally make strict mean-field assumptions. To address the above issues, we present two im- provements. First, we present a general frame- work of Bayesian logistic supervised topic models with a regularization parameter to better balance response variables and words. Technically, instead of doing standard Bayesian inference via Bayes’ rule, which requires a normalized likelihood mod- el, we propose to do regularized Bayesian infer- ence (Zhu et al., 2011; Zhu et al., 2013b) via solv- ing an optimization problem, where the posterior regularization is defined as an expectation of a l- ogistic loss, …

Content truncated to 500 words