Interpretable Entity Representations through Large-Scale Typing

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 612–624 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 612 Interpretable Entity Representations through Large-Scale Typing Yasumasa Onoe and Greg Durrett Department of Computer Science The University of Texas at Austin {yasumasa, gdurrett}@cs.utexas.edu Abstract In standard methodology for natural language processing, entities in text are typically em- bedded in dense vector spaces with pre-trained models. The embeddings produced this way are effective when fed into downstream mod- els, but they require end-task fine-tuning and are fundamentally difficult to interpret. In this paper, we present an approach to cre- ating entity representations that are human readable and achieve high performance on entity-related tasks out of the box. Our representations are vectors whose values cor- respond to posterior probabilities over fine- grained entity types, indicating the confidence of a typing model’s decision that the entity belongs to the corresponding type. We ob- tain these representations using a fine-grained entity typing model, trained either on super- vised ultra-fine entity typing data (Choi et al., 2018) or distantly-supervised examples from Wikipedia. On entity probing tasks involving recognizing entity identity, our embeddings used in parameter-free downstream models achieve competitive performance with ELMo- and BERT-based embeddings in trained mod- els. We also show that it is possible to reduce the size of our type set in a learning-based way for particular domains. Finally, we show that these embeddings can be post-hoc modified through a small number of rules to incorporate domain knowledge and improve performance. 1 Introduction In typical neural NLP systems, entities are em- bedded in the same space as other words either in context-independent (Mikolov et al., 2013; Pen- nington et al., 2014) or in context-dependent ways (Peters et al., 2018; Devlin et al., 2019). Such ap- proaches are powerful: pre-trained language mod- els implicitly learn factual knowledge about those entities (Petroni et al., 2019; Roberts et al., 2020; Jiang et al., 2020) and these representations can be grounded in structured and human-curated knowl- edge bases (Logan et al., 2019; Levine et al., 2019; Peters et al., 2019; Zhang et al., 2019; Poerner et al., 2019; Xiong et al., 2020; Wang et al., 2020). How- ever, these embeddings do not explicitly maintain representations of this knowledge, and dense en- tity representations are not directly interpretable. Knowledge probing tasks can be used to measure LMs’ factual knowledge (Petroni et al., 2019), but designing the right probing task is another hard problem (Chen et al., 2019; Poerner et al., 2019), particularly if the probes are parameter-rich (He- witt and Manning, 2019). In this work, we explore a set of interpretable entity representations that are simultaneously hu- man and machine readable. The key idea of this approach is to use fine-grained entity typing mod- els with large type inventories (Ling and Weld, 2012; Gillick et al., 2014; Choi et al., 2018; Onoe and Durrett, 2020). Given an entity mention and context words, our typing model outputs a high- dimensional vector whose values are associated with …

Content truncated to 500 words