Entities as Experts: Sparse Memory Access with Entity Supervision

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4937–4951, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4937 Entities as Experts: Sparse Memory Access with Entity Supervision Thibault F´evry ∗ thibaultfevry@gmail.com Livio Baldini Soares liviobs@google.com Nicholas FitzGerald nfitz@google.com Eunsol Choi† The University of Austin at Texas eunsol@cs.utexas.edu Tom Kwiatkowski tomkwiat@google.com Google Research Abstract We focus on the problem of capturing declar- ative knowledge about entities in the learned parameters of a language model. We introduce a new model—Entities as Experts (EAE)— that can access distinct memories of the en- tities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EAE’s entity representa- tions are learned directly from text. We show that EAE’s learned representations capture suf- ficient knowledge to answer TriviaQA ques- tions such as “Which Dr. Who villain has been played by Roger Delgado, Anthony Ain- ley, Eric Roberts?”, outperforming an encoder- generator Transformer model with 10× the pa- rameters. According to the LAMA knowledge probes, EAE contains more factual knowledge than a similarly sized BERT, as well as previ- ous approaches that integrate external sources of entity knowledge. Because EAE associates parameters with specific entities, it only needs to access a fraction of its parameters at infer- ence time, and we show that the correct identi- fication and representation of entities is essen- tial to EAE’s performance. 1 Introduction Neural network sequence models, pre-trained as language models, have recently revolutionized text understanding (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2018), and recent work has suggested that they could take the place of curated knowledge bases or textual corpora for tasks such as question answer- ing (Petroni et al., 2019; Roberts et al., 2020). In this paper, we focus on developing neural se- quence models that capture the knowledge required to answer questions about real world entities. To this end, we introduce a new model architecture ∗Work done during Google AI residency. †Work done at Google Research. Figure 1: Our model with an entity memory, applied to the open domain QA task. The red arrows shows the integration of the entity and token representations. that can access distinct and independent representa- tions of the entities mentioned in text. Unlike other efforts to inject entity specific knowledge into se- quence models (Peters et al., 2019; Zhang et al., 2019; Poerner et al., 2019) our model learns entity representations from text along with all the other model parameters. We call our model Entities as Experts (EAE), since it divides the parameter space according to entity identity. This name also reflects EAE’s similarities with the Massive Mixture of Ex- perts (Shazeer et al., 2017), as well as other work that integrates learned memory stores into sequence models (Weston et al., 2014; Lample et al., 2019). To understand the motivation for distinct and in- dependent entity representations, consider Figure 1. A traditional Transformer (Vaswani et al., 2017) needs to build an internal representation …

Content truncated to 500 words