EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8659 - 8673 December 7-11, 2022 ©2022 Association for Computational Linguistics EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexing Nora Kassner, Fabio Petroni, Mikhail Plekhanov, Sebastian Riedel, Nicola Cancedda Meta AI kassner@meta.com Abstract Existing work on Entity Linking mostly as- sumes that the reference knowledge base is complete, and therefore all mentions can be linked. In practice this is hardly ever the case, as knowledge bases are incomplete and because novel concepts arise constantly. We introduce the temporally segmented Un- known Entity Discovery and Indexing (EDIN) -benchmark where unknown entities, that is en- tities not part of the knowledge base and with- out descriptions and labeled mentions, have to be integrated into an existing entity linking system. By contrasting EDIN with zero-shot entity linking, we provide insight on the addi- tional challenges it poses. Building on dense- retrieval based entity linking, we introduce the end-to-end EDIN-pipeline that detects, clus- ters, and indexes mentions of unknown entities in context. Experiments show that indexing a single embedding per entity unifying the infor- mation of multiple mentions works better than indexing mentions independently. 1 Introduction Most existing works on Entity linking (EL) – the fundamental task of detecting mentions of entities in context and disambiguating them against a ref- erence knowledge base (KB) – assume that such KB is complete, and therefore all mentions can be linked. In practice this is hardly ever the case, as KBs are incomplete when they are created and be- cause novel concepts arise constantly. For example, English Wikipedia, often used as the reference KB for large scale linking, is growing by more than 17k entities every month.1 Consequently, at the time of deployment EL sys- tems are quickly outdated and static evaluation overestimates performance. But as these systems play significant role in many real world industry 1https://en.wikipedia.org/wiki/Wikipedia: Size_of_Wikipedia (09.05.2022) applications, e.g., moderating discussions around recent events, a dynamic look on EL is crucial. Nonetheless, related work on this problem is sparse. Available datasets (Ji et al., 2015; Der- czynski et al., 2017; Nakashole et al., 2013) and models (Hoffart et al., 2014) are outdated and/or small scale and use features which are not read- ily available (Nakashole et al., 2013; Wu et al., 2016). Most importantly, they approach the prob- lem only in parts. We revisit this problem in context of dense-retrieval and large-scale EL, e.g., EL rely- ing on bi-encoder architecture that runs a nearest neighbor search between mention encoding and a large-scale index of entity encodings. To this end, we introduce EDIN-benchmark and EDIN- pipeline where unknown entities, that is entities with no available canonical names, descriptions and labeled mentions, have to be integrated into an existing EL model in an end-to-end fashion. To the best of our knowledge, EDIN-pipeline is the first end-to-end pipeline tackling this problem. Note that this setting is strictly more demand- ing than zero-shot (zs) entity linking (Logeswaran et al., 2019), where a …

Content truncated to 500 words