E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 803–818 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 803 E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT Nina Poerner∗† and Ulli Waltinger† and Hinrich Sch¨utze∗ ∗Center for Information and Language Processing, LMU Munich, Germany †Corporate Technology Machine Intelligence (MIC-DE), Siemens AG Munich, Germany poerner@cis.uni-muenchen.de | inquiries@cislmu.org Abstract We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT’s native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E- BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pre- training of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E- BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that some- one with an Italian-sounding name speaks Ital- ian), and that E-BERT mitigates this problem. 1 Introduction BERT (Devlin et al., 2019) and its successors (e.g., Yang et al. (2019); Liu et al. (2019); Wang et al. (2019b)) continue to achieve state of the art per- formance on various NLP tasks. Recently, there has been interest in enhancing BERT with factual knowledge about entities (Zhang et al., 2019; Pe- ters et al., 2019). To this end, we introduce E- BERT: We align Wikipedia2Vec entity vectors (Ya- mada et al., 2016) with BERT’s wordpiece vector space (Section 3.1) and feed the aligned vectors into BERT as if they were wordpiece vectors (Sec- tion 3.2). Importantly, we do not make any changes to the BERT encoder itself, and we do no additional pretraining. This stands in contrast to previous entity-enhanced versions of BERT, such as ERNIE or KnowBert, which require additional encoder pre- training. In Section 4, we evaluate our approach on LAMA (Petroni et al., 2019), a recent unsupervised QA benchmark for pretrained Language Models (LMs). We set a new state of the art on LAMA, with improvements over original BERT, ERNIE and KnowBert. We also find that the original BERT model is overly reliant on the surface form of en- tity names, e.g., it predicts that a person with an Italian-sounding name speaks Italian, regardless of whether this is factually correct. To quantify this ef- fect, we create LAMA-UHN (UnHelpfulNames), a subset of LAMA where questions with overly helpful entity names were deleted (Section 4.4). In Section 5, we show how to apply E-BERT to two entity-centric downstream tasks: relation classification (Section 5.1) and entity linking (Sec- tion 5.2). On the former task, we feed aligned entity vectors as inputs, on the latter, they serve as inputs and outputs. In both cases, E-BERT outperforms original BERT and other baselines. …

Content truncated to 500 words