Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8308–8321, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8308 Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction Tara Safavi∗ University of Michigan tsafavi@umich.edu Danai Koutra University of Michigan dkoutra@umich.edu Edgar Meij Bloomberg emeij@bloomberg.net Abstract Little is known about the trustworthiness of predictions made by knowledge graph embed- ding (KGE) models. In this paper we take ini- tial steps toward this direction by investigating the calibration of KGE models, or the extent to which they output confidence scores that reflect the expected correctness of predicted knowledge graph triples. We first conduct an evaluation under the standard closed-world as- sumption (CWA), in which predicted triples not already in the knowledge graph are con- sidered false, and show that existing calibra- tion techniques are effective for KGE under this common but narrow assumption. Next, we introduce the more realistic but challenging open-world assumption (OWA), in which un- observed predictions are not considered true or false until ground-truth labels are obtained. Here, we show that existing calibration tech- niques are much less effective under the OWA than the CWA, and provide explanations for this discrepancy. Finally, to motivate the util- ity of calibration for KGE from a practitioner’s perspective, we conduct a unique case study of human-AI collaboration, showing that cali- brated predictions can improve human perfor- mance in a knowledge graph completion task. 1 Introduction Knowledge graphs are essential resources in nat- ural language processing tasks such as question answering and reading comprehension (Shen et al., 2019; Yang et al., 2019). Because they are by na- ture incomplete, extensive research efforts have been invested into completing them via different techniques (Ji et al., 2020; Belth et al., 2020). One such technique is knowledge graph embed- ding (KGE), which involves learning latent rep- resentations of entities and relations to be used toward predicting new facts. KGE models are most ∗This work was done during an internship at Bloomberg. Figure 1: An example of how optimizing for rank- ing does not necessarily lead to trustworthy prediction scores. Here, an uncalibrated KGE model would per- form well according to ranking metrics because the true triple is ranked highly, even though it receives a much lower score than the incorrect top-ranked triple and a similar score to the nonsensical triple below it. commonly optimized for the link prediction task, which tests their ability to “learn to rank” plausible knowledge graph triples higher than implausible ones. While KGE accuracy as measured by ranking- based link prediction metrics has been steadily improving on benchmark datasets over the past decade (Ruffinelli et al., 2020), such evaluation setups can be misleading. As shown in Figure 1, ranking only considers the ordering of prediction scores, so models can perform well according to ranking metrics even if they assign high scores to incorrect or nonsensical triples (Wang et al., 2019). As such, the practical utility of KGE for real- world knowledge graph completion remains lim- ited, …

Content truncated to 500 words