Model-Agnostic Bias Measurement in Link Prediction

Findings of the Association for Computational Linguistics: EACL 2023, pages 1632–1648 May 2-6, 2023 ©2023 Association for Computational Linguistics Model-Agnostic Bias Measurement in Link Prediction Lena Schwertmann Hasso Plattner Institute / University of Potsdam Potsdam, Germany lena-schwertmann@gmx.de Manoj Prabhakar Kannan Ravi∗ LexisNexis Berlin, Germany manoj.prabhakarkr91@gmail.com Gerard de Melo Hasso Plattner Institute / University of Potsdam Potsdam, Germany gerard.demelo@hpi.de Abstract Link prediction models based on factual knowledge graphs are commonly used in applications such as search and question answering. However, work investigating social bias in these models has been limited. Previous work focused on knowledge graph embeddings, so more recent classes of models achieving superior results by fine-tuning Trans- formers have not yet been investigated. We therefore present a model-agnostic approach for bias measurement leveraging fairness metrics to compare bias in knowledge graph embedding-based predictions (KG only) with models that use pre-trained, Transformer-based language models (KG+LM). We further create a dataset to measure gender bias in occupation predictions and assess whether the KG+LM models are more or less biased than KG only models. We find that gender bias tends to be higher for the KG+LM models and analyze potential connections to the accuracy of the models and the data bias inherent in our dataset. Finally, we discuss limitations and ethical considerations of our work. The repository containing the source atcode and the data set is publicly available https://github.com/lena-schwert/ comparing-bias-in-KG-models. 1 Introduction Achieving reliable link prediction in factual knowl- edge graphs (KGs) is an important goal to over- come the inherent gaps in their knowledge. Such graphs are widely used by companies such as Google, LinkedIn, Amazon, and Bloomberg across a range of different real-world applications, includ- ing search, recommender systems, and voice-based question answering (Hogan et al., 2021; Weikum et al., 2021; Ji et al., 2021). Typically, information is stored in the shape of triples (h, r, t), consisting of a head entity h, a relation r, and a tail entity t. ∗ Work conducted at Hasso-Plattner-Institute / University of Potsdam. Entities express concepts, while relations express the connection between them, e.g., (Barack Obama, occupation, politician). Link prediction models score the plausibility of a given fact, with two dis- tinct purposes: (i) They make the graph structure available to machine learning models, e.g., in the form of embeddings, (ii) and – if sufficiently reli- able – may eventually be used to make plausible predictions of missing facts, i.e., solving the prob- lem of automatic knowledge graph completion and refinement (Hogan et al., 2021; Paulheim, 2016). While link prediction models are naturally eval- uated for their accuracy, there have only recently been studies that assess possible biases that they may exhibit. Echoing prior position papers on bias in factual KGs (Janowicz et al., 2018; Kraft and Usbeck, 2022), we consider an analysis of bias as essential for a thorough model evaluation, espe- cially because a) KGs contain sensitive informa- tion about humans (e.g., gender), b) historical facts naturally contain historical biases, and c) the ap- plications of KG-based models are increasingly so- …

Content truncated to 500 words