Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6397–6407, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6397 Scalable Zero-shot Entity Linking with Dense Entity Retrieval Ledell Wu,1 Fabio Petroni,1 Martin Josifoski,3∗Sebastian Riedel,1,2 Luke Zettlemoyer1 1Facebook AI Research {ledell, fabiopetroni, sriedel, lsz}@fb.com 2University College London 3Ecole Polytechnique Federale de Lausanne martin.josifoski@epfl.ch Abstract This paper introduces a conceptually simple, scalable, and highly effective BERT-based en- tity linking model, along with an extensive evaluation of its accuracy-speed trade-off. We present a two-stage zero-shot linking algo- rithm, where each entity is defined only by a short textual description. The first stage does retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions. Each candidate is then re-ranked with a cross- encoder, that concatenates the mention and en- tity text. Experiments demonstrate that this approach is state of the art on recent zero- shot benchmarks (6 point absolute gains) and also on more established non-zero-shot eval- uations (e.g. TACKBP-2010), despite its rel- ative simplicity (e.g. no explicit entity em- beddings or manually engineered mention ta- bles). We also show that bi-encoder link- ing is very fast with nearest neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the ac- curacy gain from the more expensive cross- encoder can be transferred to the bi-encoder via knowledge distillation. Our code and models are available at https://github. com/facebookresearch/BLINK. 1 Introduction Scale is a key challenge for entity linking; there are millions of possible entities to consider for each mention. To efficiently filter or rank the candi- dates, existing methods use different sources of external information, including manually curated mention tables (Ganea and Hofmann, 2017), incom- ing Wikipedia link popularity (Yamada et al., 2016), and gold Wikipedia entity categories (Gillick et al., 2019). In this paper, we show that BERT-based models set new state-of-the-art performance levels ∗Work done during internship with Facebook. for large scale entity linking when used in a zero shot setup, where there is no external knowledge and a short text description provides the only infor- mation we have for each entity. We also present an extensive evaluation of the accuracy-speed trade- off inherent to large pre-trained models, and show is possible to achieve very efficient linking with modest loss of accuracy. More specifically, we introduce a two stage ap- proach for zero-shot linking (see Figure 1 for an overview), based on fine-tuned BERT architectures (Devlin et al., 2019). In the first stage, we do re- trieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions (Humeau et al., 2019; Gillick et al., 2019). Each retrieved candidate is then ex- amined more carefully with a cross-encoder that concatenates the mention and entity text, follow- ing Logeswaran et al. (2019). This overall approach is conceptually simple but highly effective, as we show through detailed experiments. Our two-stage approach achieves a new state-of- the-art result on TACKBP-2010, with …

Content truncated to 500 words