Zero-shot Entity Linking with Efficient Long Range Sequence Modeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2517–2522 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2517 Zero-shot Entity Linking with Efficient Long Range Sequence Modeling Zonghai Yao University of Massachusetts, Amherst zonghaiyao@cs.umass.edu Liangliang Cao University of Massachusetts, Amherst llcao@cs.umass.edu Huapu Pan Google Research huapupan@google.com Abstract This paper considers the problem of zero-shot entity linking, in which a link in the test time may not present in training. Following the prevailing BERT-based research efforts, we find a simple yet effective way is to ex- pand the long-range sequence modeling. Un- like many previous methods, our method does not require expensive pre-training of BERT with long position embeddings. Instead, we propose an efficient position embeddings ini- tialization method called Embedding-repeat, which initializes larger position embeddings based on BERT-Base. On Wikia’s zero-shot EL dataset, our method improves the SOTA from 76.06% to 79.08%, and for its long data, the corresponding improvement is from 74.57% to 82.14%. Our experiments suggest the effectiveness of long-range sequence mod- eling without retraining the BERT model. 1 1 Introduction Entity linking (EL) is the task of grounding en- tity mentions by linking them to entries in a given database or dictionary of entities. Traditional EL approaches often assume that entities linked at the test time are present in the training set. Neverthe- less, many real-world applications prefer the zero- shot setting, where there is no external knowledge and a short text description provides the only in- formation we have for each entity (Sil et al., 2012; Wang et al., 2015). For zero-shot entity linking (Lo- geswaran et al., 2019), it is crucial to consider the context of entity description and mention, so that the system can generalize to unseen entities. How- ever, most of the BERT-based models are based on a context window with 512 tokens, limited to capturing the long-range of context. This paper de- fines a model’s Effective-Reading-Length (ER- Length) as the total length of the mention contexts 1Our code are publicly available. https://github. com/seasonyao/Zero-Shot-Entity-Linking. Figure 1: Only models with large ERLength can solve this entity linking problem because only they can get valuable critical information in the mention contexts and entity description. and entity description that it can read. Figure 1 demonstrates an example where long ERlengths are more preferred than short ones. Many existing methods that can be used to expand ERLength (Sohoni et al., 2019; Dai et al., 2019), however, often need to completely re-do pre-training with the masked language modeling objective on the vast general corpus (like Wikipedia), which is not only very expensive but also impossible in many scenarios. This paper proposes a practical way, Embeddings-repeat, to expand BERT’s ER- Length by initializing larger position embeddings, allowing reading all information in the context. Note our method differs from previous works since it can directly use the larger position embeddings initialized from BERT-Base to do fine-tuning on downstream tasks without any retraining. Extensive experiments are conducted to compare different ways of expanding ERLength, and the results …

Content truncated to 500 words