Conceptualizing Treatment Leakage in Text-based Causal Inference

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5638 - 5645 July 10-15, 2022 ©2022 Association for Computational Linguistics Conceptualizing Treatment Leakage in Text-based Causal Inference Adel Daoud1,3,4 Connor T. Jerzak1,3 Richard Johansson2,4 1Linköping University, 2University of Gothenburg, 3Harvard University, 4Chalmers University of Technology {adel.daoud, connor.jerzak}@liu.se, richard.johansson@gu.se Abstract Causal inference methods that control for text- based confounders are becoming increasingly important in the social sciences and other dis- ciplines where text is readily available. How- ever, these methods rely on a critical assump- tion that there is no treatment leakage: that is, the text only contains information about the confounder and no information about treat- ment assignment. When this assumption does not hold, methods that control for text to ad- just for confounders face the problem of post- treatment (collider) bias. However, the as- sumption that there is no treatment leakage may be unrealistic in real-world situations in- volving text, as human language is rich and flexible. Language appearing in a public pol- icy document or health records may refer to the future and the past simultaneously, and thereby reveal information about the treatment assignment. In this article, we define the treatment-leakage problem, and discuss the identification as well as the estimation challenges it raises. Sec- ond, we delineate the conditions under which leakage can be addressed by removing the treatment-related signal from the text in a pre- processing step we define as text distillation. Lastly, using simulation, we show how treat- ment leakage introduces a bias in estimates of the average treatment effect (ATE) and how text distillation can mitigate this bias. 1 Introduction In observational settings, scholars need to collect information about potential confounders in order to estimate the causal effect (τ) of a treatment on an outcome (Daoud and Dubhashi, 2020). If we observed the set of confounders directly, we could condition on those quantities to recover unbiased causal effects. Yet, because some confounders U are difficult to measure directly, scholars are turn- ing to alternative data sources, such as medical records, policy documents, or social media posts, to indirectly measure (proxy) confounders (Kino et al., 2021). Recent methodological frameworks supply ways of integrating high-dimensional text data into causal estimation (Mozer et al., 2020; Roberts et al., 2020; Feder et al., 2021). However, prior literature has primarily assumed that documents only contain information about the confounder, but not about the treatment— something we term the no-treatment-leakage as- sumption. Here, “contain information” means that the text is caused by the treatment (or the con- founder) directly or indirectly. When treatment leakage occurs after treatment assignment, its bias is equivalent to a post-treatment bias (Pearl, 2015). Treatment leakage leads to an identification chal- lenge. The challenge is that W is both necessary for adjusting (as it is a proxy) yet it is also a post- treatment variable. Without treatment leakage, W would not be a post-treatment variable, as it does not harbour information about the treatment as- …

Content truncated to 500 words