Global Entity Disambiguation with BERT

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3264 - 3271 July 10-15, 2022 ©2022 Association for Computational Linguistics Global Entity Disambiguation with BERT Ikuya Yamada†,‡ Koki Washio§∗ Hiroyuki Shindo¶,‡ Yuji Matsumoto‡ †Studio Ousia ‡RIKEN §Megagon Labs ¶NAIST ikuya@ousia.jp kwashio@megagon.ai shindo@is.naist.jp yuji.matsumoto@riken.jp Abstract We propose a global entity disambiguation (ED) model based on BERT (Devlin et al., 2019). To capture global contextual informa- tion for ED, our model treats not only words but also entities as input tokens, and solves the task by sequentially resolving mentions to their referent entities and using resolved entities as inputs at each step. We train the model using a large entity-annotated corpus obtained from Wikipedia. We achieve new state-of-the-art results on five standard ED datasets: AIDA-CoNLL, MSNBC, AQUAINT, ACE2004, and WNED-WIKI. The source code and model checkpoint are available at https: //github.com/studio-ousia/luke. 1 Introduction Entity disambiguation (ED) refers to the task of assigning mentions in a document to correspond- ing entities in a knowledge base (KB). This task is challenging because of the ambiguity between men- tions (e.g., “World Cup”) and the entities they refer to (e.g., FIFA World Cup or Rugby World Cup). ED models typically rely on local contextual infor- mation based on words that co-occur with the men- tion and global contextual information based on the entity-based coherence of the disambiguation decisions. A key to improve the performance of ED is to effectively combine both local and global con- textual information (Ganea and Hofmann, 2017; Le and Titov, 2018). In this study, we propose a global ED model based on BERT (Devlin et al., 2019). Our model treats words and entities in the document as in- put tokens, and is trained by predicting randomly masked entities in a large entity-annotated corpus obtained from Wikipedia. This training enables the model to learn how to disambiguate masked entities based on words and non-masked entities. At the inference time, our model disambiguates ∗ Work done at RIKEN. [MASK] [MASK] Lionel Messi [CLS] played in the world cup Input: Messi played in the World Cup mess ##i [SEP] Lionel Messi [MASK] FIFA World Cup Transformer Words Entities Transformer Lionel Messi Result: [CLS] played in the world cup mess ##i [SEP] Figure 1: The inference procedure of our model with the input text “Messi played in the World Cup.” Given mentions (“Messi” and “World Cup”), our model se- quentially resolves them to their referent entities, and uses the resolved entities as contexts at each step. mentions sequentially using words and already re- solved entities (see Figure 1). This sequential infer- ence effectively accumulates the global contextual information and enhances the coherence of disam- biguation decisions (Yang et al., 2019). We conducted extensive experiments using six standard ED datasets, i.e., AIDA-CoNLL, MSNBC, AQUAINT, ACE2004, WNED-WIKI, and WNED-CWEB. As a result, the global con- textual information consistently improved the per- formance. Furthermore, we achieved new state of the art on all datasets except for WNED- CWEB. …

Content truncated to 500 words