Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond

Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond Amir Feder1,10∗, Katherine A. Keith2, Emaad Manzoor3, Reid Pryzant4, Dhanya Sridhar5, Zach Wood-Doughty6, Jacob Eisenstein7, Justin Grimmer8, Roi Reichart1, Margaret E. Roberts9, Brandon M. Stewart10, Victor Veitch7,11, and Diyi Yang12 1Technion - Israel Institute of Technology, Israel 2Williams College, USA 3University of Wisconsin - Madison, USA 4Microsoft, USA 5Columbia University, Canada 6Northwestern University, USA 7Google Research, USA 8Stanford University, USA 9University of California San Diego, USA 10Princeton University, USA 11University of Chicago, USA 12Georgia Tech, USA Abstract A fundamental goal of scientific research is to learn about causal relationships. However, despite its critical role in the life and so- cial sciences, causality has not had the same importance in Natural Language Processing (NLP), which has traditionally placed more emphasis on predictive tasks. This distinc- tion is beginning to fade, with an emerging area of interdisciplinary research at the con- vergence of causal inference and language processing. Still, research on causality in NLP remains scattered across domains without uni- fied definitions, benchmark datasets and clear articulations of the challenges and opportuni- ties in the application of causal inference to the textual domain, with its unique proper- ties. In this survey, we consolidate research across academic areas and situate it in the broader NLP landscape. We introduce the sta- tistical challenge of estimating causal effects with text, encompassing settings where text is used as an outcome, treatment, or to address confounding. In addition, we explore potential uses of causal inference to improve the ro- bustness, fairness, and interpretability of NLP ∗All authors equally contributed to this paper. Au- thor names are organized alphabetically in two clusters: First students and post-docs and then faculty members. The email address of the corresponding (first) author is: feder@campus.technion.ac.il. models. We thus provide a unified overview of causal inference for the NLP community.1 1 Introduction The increasing effectiveness of NLP has created exciting new opportunities for interdisciplinary collaborations, bringing NLP techniques to a wide range of external research disciplines (e.g., Roberts et al., 2014; Zhang et al., 2020; Ophir et al., 2020) and incorporating new data and tasks into mainstream NLP (e.g., Thomas et al., 2006; Pryzant et al., 2018). In such interdisciplinary collaborations, many of the most important re- search questions relate to the inference of causal relationships. For example, before recommending a new drug therapy, clinicians want to know the causal effect of the drug on disease progression. Causal inference involves a question about a coun- terfactual world created by taking an intervention: What would a patient’s disease progression have been if we had given them the drug? As we ex- plain below, with observational data, the causal effect is not equivalent to the correlation between whether the drug is taken and the observed dis- ease progression. There is now a vast literature on techniques for making valid inferences using 1An online repository containing existing research on causal inference and language processing is available here: https://github.com/causaltext/causal -text-papers. 1138 Transactions of the Association for Computational …

Content truncated to 500 words