Learning Faithful Representations of Causal Graphs

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 839–850 August 1–6, 2021. ©2021 Association for Computational Linguistics 839 Learning Faithful Representations of Causal Graphs Ananth Balashankar Lakshminarayanan Subramanian New York University New York, NY, USA {ananth,lakshmi}@nyu.edu Abstract Learning contextual text embeddings that rep- resent causal graphs has been useful in improv- ing the performance of downstream tasks like causal treatment effect estimation. However, existing causal embeddings which are trained to predict direct causal links, fail to capture other indirect causal links of the graph, thus leading to spurious correlations in downstream tasks. In this paper, we define the faithful- ness property of contextual embeddings to cap- ture geometric distance-based properties of di- rected acyclic causal graphs. By incorporating these faithfulness properties, we learn text em- beddings that are 31.3% more faithful to hu- man validated causal graphs with about 800K and 200K causal links and achieve 21.1% bet- ter Precision-Recall AUC in a link prediction fine-tuning task. Further, in a crowdsourced causal question-answering task on Yahoo! An- swers with questions of the form “What causes X?”, our faithful embeddings achieved a pre- cision of the first ranked answer (P@1) of 41.07%, outperforming the existing baseline by 10.2%. 1 Introduction Learning distributed word representations that cap- ture causal relationships are useful for real-world natural language processing tasks (Roberts et al., 2020; Veitch et al., 2020; Gao et al., 2018, 2019). Approximating the notion of causality with a similarity-based distance metric using separate vec- tor representations for cause and effect tokens has led to significant improvement in the performance of downstream tasks like Question Answering, but can be too restrictive to generalize over unobserved edges in larger causal graphs (Sharp et al., 2016). In downstream causal reasoning based tasks like dialog systems (Ning et al., 2018), explanation gen- eration (Grimsley et al., 2020), question answering (Sharp et al., 2016), it is important to align the models with the corresponding causal graph. How- ever, words that have low cosine similarity capture various semantic similarities, like relatedness, syn- onyms, replaceability, or complementarity, but not directionality (Hamilton et al., 2017). Hence, any symmetric distance in an embedding space cannot convey the directed causal semantics for a down- stream task (M´emoli et al., 2016). In this paper, we overcome these two shortcomings and propose to optimize for directed faithfulness (Spirtes et al., 1993) that word embeddings have to satisfy towards a causal graph. Prior work on capturing sufficient information for causal inference tasks from embeddings aims to directly use them for average treatment effect estimation (Veitch et al., 2020). We are, however, interested in a complementary question: “Can we learn word embeddings based on a distance mea- sure that maps the directed distance between nodes in a causal graph to that in the embedding space?”. Unlike prior work, which aims to learn a causal aware embedding restricted to direct link predic- tion (Hamilton et al., 2017), we propose faithful- ness constraints …

Content truncated to 500 words