LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6442–6454, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6442 LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention Ikuya Yamada1,2 ikuya@ousia.jp Akari Asai3 akari@cs.washington.edu Hiroyuki Shindo4,2 shindo@is.naist.jp Hideaki Takeda5 takeda@nii.ac.jp Yuji Matsumoto2 matsu@is.naist.jp 1Studio Ousia 2RIKEN AIP 3University of Washington 4Nara Institute of Science and Technology 5National Institute of Informatics Abstract Entity representations are useful in natural lan- guage tasks involving entities. In this paper, we propose new pretrained contextualized rep- resentations of words and entities based on the bidirectional transformer (Vaswani et al., 2017). The proposed model treats words and entities in a given text as independent to- kens, and outputs contextualized representa- tions of them. Our model is trained using a new pretraining task based on the masked lan- guage model of BERT (Devlin et al., 2019). The task involves predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also propose an entity-aware self-attention mecha- nism that is an extension of the self-attention mechanism of the transformer, and consid- ers the types of tokens (words or entities) when computing attention scores. The pro- posed model achieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains state-of-the-art results on five well-known datasets: Open En- tity (entity typing), TACRED (relation classi- fication), CoNLL-2003 (named entity recog- nition), ReCoRD (cloze-style question an- swering), and SQuAD 1.1 (extractive ques- tion answering). Our source code and pre- trained representations are available at https: //github.com/studio-ousia/luke. 1 Introduction Many natural language tasks involve entities, e.g., relation classification, entity typing, named entity recognition (NER), and question answering (QA). Key to solving such entity-related tasks is a model to learn the effective representations of entities. Conventional entity representations assign each en- tity a fixed embedding vector that stores informa- tion regarding the entity in a knowledge base (KB) (Bordes et al., 2013; Trouillon et al., 2016; Yamada et al., 2016, 2017). Although these models capture the rich information in the KB, they require entity linking to represent entities in a text, and cannot represent entities that do not exist in the KB. By contrast, contextualized word representa- tions (CWRs) based on the transformer (Vaswani et al., 2017), such as BERT (Devlin et al., 2019), and RoBERTa (Liu et al., 2020), provide effec- tive general-purpose word representations trained with unsupervised pretraining tasks based on lan- guage modeling. Many recent studies have solved entity-related tasks using the contextualized rep- resentations of entities computed based on CWRs (Zhang et al., 2019; Peters et al., 2019; Joshi et al., 2020). However, the architecture of CWRs is not well suited to representing entities for the follow- ing two reasons: (1) Because CWRs do not out- put the span-level representations of entities, they typically need to learn how to compute such rep- resentations based on a downstream dataset that is typically small. (2) Many entity-related tasks, e.g., relation classification and QA, involve reasoning about the relationships between entities. Although …

Content truncated to 500 words