Biomedical Interpretable Entity Representations

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3547–3561 August 1–6, 2021. ©2021 Association for Computational Linguistics 3547 Biomedical Interpretable Entity Representations Diego Garcia-Olano1,2 Yasumasa Onoe2 Ioana Baldini1 Joydeep Ghosh2 Byron C. Wallace3 Kush R. Varshney1 1IBM Research, 2University of Texas at Austin, 3Northeastern University diegoolano@gmail.com, yasumasa@utexas.edu, ioana@us.ibm.com, ghosh@ece.utexas.edu, b.wallace@northeastern.edu, krvarshn@us.ibm.com Abstract Pre-trained language models induce dense entity representations that offer strong per- formance on entity-centric NLP tasks, but such representations are not immediately inter- pretable. This can be a barrier to model uptake in important domains such as biomedicine. There has been recent work on general in- terpretable representation learning (Onoe and Durrett, 2020), but these domain-agnostic rep- resentations do not readily transfer to the im- portant domain of biomedicine. In this paper, we create a new entity type system and train- ing set from a large corpus of biomedical texts by mapping entities to concepts in a medical ontology, and from these to Wikipedia pages whose categories are our types. From this map- ping we derive Biomedical Interpretable En- tity Representations (BIERs), in which dimen- sions correspond to fine-grained entity types, and values are predicted probabilities that a given entity is of the corresponding type. We propose a novel method that exploits BIER’s final sparse and intermediate dense representa- tions to facilitate model and entity type debug- ging. We show that BIERs achieve strong per- formance in biomedical tasks including named entity disambiguation and entity label classifi- cation, and we provide error analysis to high- light the utility of their interpretability, partic- ularly in low-supervision settings. Finally, we provide our induced 68K biomedical type sys- tem, the corresponding 37 million triples of de- rived data used to train BIER models and our best performing model. 1 Introduction In modern NLP systems, entities are embedded in the same dense vector space as words using vec- tors from pre-trained (masked) language models (Devlin et al., 2019) that yield contextualized em- beddings of tokens. These representations are used as inputs for downstream models built for particular tasks. One issue with such learned representations is that we do not actually know what information they encode. Recent work has shown that deep pre-trained models implicitly learn factual knowl- edge about entities (Petroni et al., 2019; Roberts et al., 2020), but the embeddings that they provide do not explicitly maintain representations of this knowledge (i.e., the dimensions in learned represen- tations have no a priori semantics); consequently, are not directly interpretable. This has motivated the design of knowledge probing tasks to measure a factual knowledge implicit in embeddings (Petroni et al., 2019; Poerner et al., 2019). Recent work (Onoe and Durrett, 2020) has pro- posed learning interpretable entity representations using an entity typing model and corresponding fine-grained type system that accepts an entity men- tion and its context. The output represents a high- dimensional sparse embedding whose values cor- respond to the model’s (independently) predicted probabilities that the entity possesses the respective properties defined by the fine-grained type system. This past work …

Content truncated to 500 words