Heterogeneous Supervised Topic Models

Heterogeneous Supervised Topic Models Dhanya Sridhar⋄ and Hal Daum´e III† and David Blei ♠ ⋄Universit´e de Montr´eal and Mila-Quebec AI Institute, Canada dhanya.sridhar@mila.quebec †University of Maryland and Microsoft Research, USA hal3@umd.edu ♠Columbia University, USA david.blei@columbia.edu Abstract Researchers in the social sciences are often in- terested in the relationship between text and an outcome of interest, where the goal is to both uncover latent patterns in the text and predict outcomes for unseen texts. To this end, this paper develops the heterogeneous supervised topic model (HSTM), a probabilistic approach to text analysis and prediction. HSTMs posit a joint model of text and outcomes to find heterogeneous patterns that help with both text analysis and prediction. The main benefit of HSTMs is that they capture heterogeneity in the relationship between text and the outcome across latent topics. To fit HSTMs, we de- velop a variational inference algorithm based on the auto-encoding variational Bayes frame- work. We study the performance of HSTMs on eight datasets and find that they consis- tently outperform related methods, including fine-tuned black-box models. Finally, we ap- ply HSTMs to analyze news articles labeled with pro- or anti-tone. We find evidence of differing language used to signal a pro- and anti-tone. 1 Introduction Researchers in the social sciences are interested in modeling the relationship between text and an outcome of interest. In surveys about elections, how do respondents’ open-ended responses relate to evaluations of their anger, fear, or concern (Roberts et al., 2014)? In news media, how do articles relate to the tone used by the writer towards the article’s topic (Card et al., 2015)? On social media, how do tweets on mass shootings relate to the political ideology of the user (Demszky et al., 2019)? In these settings, the goal is to both uncover latent properties about the text and predict outcomes for unseen texts. We develop the heterogeneous supervised topic model (HSTM), a probabilistic approach to mod- eling outcomes from text. The key innovation in HSTMs—the hetereogeneity—is the idea that in- dividual words can be predictive of the outcome not just on their own, but also specifically in combination with latent topics in the text. As a running example, consider news articles about US immigration labeled with the pro- or anti-tone of the writer towards US immigration (Card et al., 2015). Table 1 illustrates how an HSTM can be used to analyze patterns in text that relate to pro- and anti-immigration tones. As is standard for topic models (Blei et al., 2003), HSTMs learn the hidden topics in the cor- pus, which are captured by the ‘‘neutral’’ words shown in each topic. For example, US immigra- tion articles discuss legal aspects (first topic) and views of the administration (second topic). However, HSTMs also analyze the per-topic, heterogeneous associations between the text and outcome. In Table 1, this is captured by the ‘‘pro’’ and ‘‘anti’’ words in each topic. For example, in articles about the former administration (second topic), phrases such as ‘‘families’’ and ‘‘work- ers’’ reflect a …

Content truncated to 500 words