Robustness Evaluation of Entity Disambiguation Using Prior Probes: the Case of Entity Overshadowing

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10501–10510 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10501 Robustness Evaluation of Entity Disambiguation Using Prior Probes: the Case of Entity Overshadowing Vera Provatorova, Samarth Bhargav, Svitlana Vakulenko, Evangelos Kanoulas University of Amsterdam, Amsterdam, The Netherlands {v.provatorova, s.bhargav, s.vakulenko, e.kanoulas}@uva.nl Abstract Entity disambiguation (ED) is the last step of entity linking (EL), when candidate entities are reranked according to the context they ap- pear in. All datasets for training and evaluating models for EL consist of convenience samples, such as news articles and tweets, that prop- agate the prior probability bias of the entity distribution towards more frequently occurring entities. It was previously shown that perfor- mance of EL systems on such datasets is over- estimated, since it is possible to obtain higher accuracy scores by merely learning the prior. To provide a more adequate evaluation bench- mark, we introduce the ShadowLink dataset, which includes 16K short text snippets anno- tated with entity mentions. We evaluate and report the performance of several popular EL systems on the ShadowLink benchmark. The results show a considerable difference in ac- curacy between common and uncommon am- biguous entities that require disambiguation, for all of the EL systems under evaluation, demonstrating the effects of prior probability bias and entity overshadowing. 1 Introduction The task of entity linking (EL) refers to finding named entity mentions in unstructured documents and matching them with the corresponding entries in a structured knowledge graph (Milne and Wit- ten, 2008; Oliveira et al., 2021). This matching is usually done using the surface form of an entity, which is a text label assigned to an entity in the knowledge graph (van Hulst et al., 2020). Some mentions may have several possible matches: for example, “Michael Jordan” may refer either to a well-known scientist or the basketball player, since they share the same surface form. Such mentions are ambiguous and require an additional step of entity disambiguation (ED), which is conditioned on the context in which the mentions appear in the text, to be linked correctly. Following van Erp and Groth (2020) we refer to a set of entities that share the same surface form as an entity space. (a) “Michael Jordan” (scientist) is overshadowed by “Michael Jordan” (basketball player). (b) Even with more relevant context, overshadowing persists. Figure 1: An example of entity overshadowing. The correct entity is ranked lower by the EL systems (indi- cated in blue) than the more common one. To decide which of the possible matches is the correct one, an ED algorithm typically relies on: (1) contextual similarity, which is derived from the document in which the mention appears, in- dicating the relatedness of the candidate entity to the document content, and (2) entity importance, which is the prior probability of encountering the candidate entity irrespective of the document con- tent, indicating its commonness (Milne and Witten, 2008; Ferragina and Scaiella, 2012; van Hulst et al., 2020). The standard datasets currently used …

Content truncated to 500 words