Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 677–685 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 677 Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking Samuel Broscheit Data and Web Science Group, University of Mannheim, Germany broscheit@informatik.uni-mannheim.de Abstract A typical architecture for end-to-end entity linking systems consists of three steps: men- tion detection, candidate generation and en- tity disambiguation. In this study we inves- tigate the following questions: (a) Can all those steps be learned jointly with a model for contextualized text-representations, i.e. BERT (Devlin et al., 2019)? (b) How much entity knowledge is already contained in pretrained BERT? (c) Does additional entity knowledge improve BERT’s performance in downstream tasks? To this end, we propose an extreme simplification of the entity linking setup that works surprisingly well: simply cast it as a per token classification over the entire entity vocabulary (over 700K classes in our case). We show on an entity linking benchmark that (i) this model improves the entity representa- tions over plain BERT, (ii) that it outperforms entity linking architectures that optimize the tasks separately and (iii) that it only comes second to the current state-of-the-art that does mention detection and entity disambiguation jointly. Additionally, we investigate the use- fulness of entity-aware token-representations in the text-understanding benchmark GLUE, as well as the question answering benchmarks SQUAD V2 and SWAG and also the EN-DE WMT14 machine translation benchmark. To our surprise, we find that most of those bench- marks do not benefit from additional entity knowledge, except for a task with very small training data, the RTE task in GLUE, which improves by 2%. 1 Introduction The goal of entity linking is, given a knowledge base (KB) and unstructured data, e.g. text, to detect mentions of the KB’s entities in the unstructured data and link them to the correct KB entry. The entity linking task is typically implemented by the following steps: • Mention detection (MD): text spans of poten- tial entity mentions are identified, • Candidate generation (CG): entity candidates for each mention are retrieved from the KB, • Entity disambiguation (ED): (typically) a mix of useful coreference and coherence features together with a classifier determine the entity link. Durrett and Klein (2014) were the first to propose jointly modelling MD, CG and ED in a graphical model and could show that each of those steps are interdependent and benefit from a joint objective. Other approaches only model MD and ED jointly (Nguyen et al., 2016; Kolitsas et al., 2018), thus these architectures depend on a CG step after men- tion detection. Hachey et al. (2013); Guo et al. (2013); Durrett and Klein (2014) showed the influ- ence of CG on entity linking, because it can be the coverage bottleneck, when the correct entity is not contained in the candidates for ED. Yamada et al. (2016, 2017) use a precomputed set of entity candi- dates published by Pershina et al. (2015) for their experiments on the CoNLL03/AIDA benchmark …

Content truncated to 500 words