Highly Parallel Autoregressive Entity Linking with Discriminative Correction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7662–7669 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7662 Highly Parallel Autoregressive Entity Linking with Discriminative Correction Nicola De Cao 1,2, Wilker Aziz 1, Ivan Titov 1,2 1University of Amsterdam, 2University of Edinburgh { nicola.decao, w.aziz, titov } @uva.nl Abstract Generative approaches have been recently shown to be effective for both Entity Disam- biguation and Entity Linking (i.e., joint men- tion detection and disambiguation). However, the previously proposed autoregressive formu- lation for EL suffers from i) high computa- tional cost due to a complex (deep) decoder, ii) non-parallelizable decoding that scales with the source sequence length, and iii) the need for training on a large amount of data. In this work, we propose a very efficient approach that parallelizes autoregressive linking across all potential mentions and relies on a shallow and efficient decoder. Moreover, we augment the generative objective with an extra discrimi- native component, i.e. a correction term which lets us directly optimize the generator’s rank- ing. When taken together, these techniques tackle all the above issues: our model is >70 times faster and more accurate than the pre- vious generative method, outperforming state- of-the-art approaches on the standard English dataset AIDA-CoNLL. 1 1 Introduction Entity Linking (EL; Bunescu and Pa¸sca, 2006; Cucerzan, 2007; Dredze et al., 2010; Hoffart et al., 2011; Le and Titov, 2018) is a fundamental task in NLP employed as a building block for text under- standing (Févry et al., 2020a; Verga et al., 2020), question answering (Nie et al., 2019; Asai et al., 2020; De Cao et al., 2019), dialog modeling (Di- nan et al., 2019; Sevegnani et al., 2021), and in- formation extraction (Sarawagi, 2008; Martinez- Rodriguez et al., 2020), to name a few. Popular ear- lier methods address the Mention Detection (MD) and Entity Disambiguation (ED) stages of EL sep- arately (Ceccarelli et al., 2013; Daiber et al., 2013; Steinmetz and Sack, 2013; Piccinno and Ferragina, 1Source code available at https://github.com/ nicola-decao/efficient-autoregressive-EL 2014) while modern techniques leverage their mu- tual dependency (Kolitsas et al., 2018; Broscheit, 2019; Martins et al., 2019). A new line of work (De Cao et al., 2021a,b) departs from linking mentions using a vector space and instead uses large lan- guage models fine-tuned with a generative objec- tive (i.e., predicting a textual identifier as the entity identifier). Employing autoregressive language models bet- ter leverages the implicit knowledge accumulated during pre-training, exploiting a full cross-encoder of entities and their context. For ED, autoregressive generation is remarkably good (even in multilin- gual settings), while for EL, although state-of-the- art on multiple datasets, it suffers from several and critical limitations. The generative model of De Cao et al. (2021a) outputs a version of the input document which is markup-annotated with men- tions linked to their respective entities. This neces- sitates using an autoregressive decoder, precluding parallelism across mentions. Generation also has a high computational cost due to relying on a com- plex and deep Transformer (Vaswani et …

Content truncated to 500 words