Discovery of Treatments from Text Corpora

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1600–1609, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Discovery of Treatments from Text Corpora Christian Fong Stanford University 655 Serra Street Stanford, CA 94305, USA christianfong@stanford.edu Justin Grimmer Stanford University 616 Serra Street Stanford, CA 94305, USA jgrimmer@stanford.edu Abstract An extensive literature in computational social science examines how features of messages, advertisements, and other cor- pora affect individuals’ decisions, but these analyses must specify the relevant features of the text before the experiment. Automated text analysis methods are able to discover features of text, but these meth- ods cannot be used to obtain the estimates of causal effects—the quantity of inter- est for applied researchers. We introduce a new experimental design and statistical model to simultaneously discover treat- ments in a corpora and estimate causal ef- fects for these discovered treatments. We prove the conditions to identify the treat- ment effects of texts and introduce the su- pervised Indian Buffet process to discover those treatments. Our method enables us to discover treatments in a training set us- ing a collection of texts and individuals’ responses to those texts, and then esti- mate the effects of these interventions in a test set of new texts and survey respon- dents. We apply the model to an exper- iment about candidate biographies, recov- ering intuitive features of voters’ decisions and revealing a penalty for lawyers and a bonus for military service. 1 Introduction Computational social scientists are often inter- ested in inferring how blocks of text, such as mes- sages from political candidates or advertising con- tent, affect individuals’ decisions (Ansolabehere and Iyengar, 1995; Mutz, 2011; Tomz and Weeks, 2013). To do so, they typically attempt to estimate the causal effect of the text: they model the out- come of interest, Y , as a function of the block of text presented to the respondent, t, and define the treatment effect of t relative to some other block of text t′ as Y (t) − Y (t′) (Rubin, 1974; Hol- land, 1986). For example, in industrial contexts researchers design A/B tests to compare two po- tential texts for a use case. Academic researchers often design one text that has a feature of inter- est and another text that lacks that feature but is otherwise identical (for example, (Albertson and Gadarian, 2015)). Both kinds of experiments as- sume researchers already know the features of text to vary and offer little help to researchers who would like to discover the features to vary. Topic models and related methods can discover important features in corpora of text data, but they are constructed in a way that makes it difficult to use the discovered features to estimate causal effects (Blei et al., 2003). Consider, for exam- ple, supervised latent Dirichlet allocation (sLDA) (Mcauliffe and Blei, 2007). It associates a topic- prevalence vector, θ, with each document where the estimated topics depend upon both the con- tent of documents and a label …

Content truncated to 500 words