Causal Effects of Linguistic Properties

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4095–4109 June 6–11, 2021. ©2021 Association for Computational Linguistics 4095 Causal Effects of Linguistic Properties Reid Pryzant1 Dallas Card1 Dan Jurafsky1 Victor Veitch2 Dhanya Sridhar3 1Stanford Unversity 2University of Chicago 3Columbia University 1{rpryzant,dcard,jurafsky}@stanford.edu 2victorveitch@gmail.com 3ds3778@columbia.edu Abstract We consider the problem of using observa- tional data to estimate the causal effects of lin- guistic properties. For example, does writing a complaint politely lead to a faster response time? How much will a positive product re- view increase sales? This paper addresses two technical challenges related to the problem be- fore developing a practical method. First, we formalize the causal quantity of interest as the effect of a writer’s intent, and establish the as- sumptions necessary to identify this from ob- servational data. Second, in practice, we only have access to noisy proxies for the linguistic properties of interest—e.g., predictions from classifiers and lexicons. We propose an esti- mator for this setting and prove that its bias is bounded when we perform an adjustment for the text. Based on these results, we in- troduce TEXTCAUSE, an algorithm for esti- mating causal effects of linguistic properties. The method leverages (1) distant supervision to improve the quality of noisy proxies, and (2) a pre-trained language model (BERT) to adjust for the text. We show that the proposed method outperforms related approaches when estimating the effect of Amazon review senti- ment on semi-simulated sales figures. Finally, we present an applied case study investigating the effects of complaint politeness on bureau- cratic response times. 1 Introduction Social scientists have long been interested in the causal effects of language, studying questions like: • How should political candidates describe their personal history to appeal to voters (Fong and Grimmer, 2016)? • How can business owners write product de- scriptions to increase sales on e-commerce platforms (Pryzant et al., 2017, 2018a)? • How can consumers word their complaints to receive faster responses (Egami et al., 2018)? • What conversational strategies can mental health counselors use to have more successful counseling sessions (Zhang et al., 2020)? To study the causal effects of linguistic prop- erties, we must reason about interventions: what would the response time for a complaint be if we could make that complaint polite while keeping all other properties (topic, pragmatics, etc.) fixed? Although it is sometimes feasible to run such ex- periments where text is manipulated and outcomes are recorded (Grimmer and Fong, 2020), analysts typically have observational data consisting of texts and outcomes obtained without intervention. This paper formalizes the estimation of causal effects of linguistic properties in observational settings. Estimating causal effects from observational data requires addressing two challenges. First, we need to formalize the causal effect of interest by specifying the hypothetical intervention to which it corresponds. The first contribution of this pa- per is articulating the causal effects of linguistic properties; we imagine intervening on the writer of a text document and telling …

Content truncated to 500 words