Self-Alignment Pretraining for Biomedical Entity Representations

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4228–4238 June 6–11, 2021. ©2021 Association for Computational Linguistics 4228 Self-Alignment Pretraining for Biomedical Entity Representations Fangyu Liu♣, Ehsan Shareghi♦,♣, Zaiqiao Meng♣, Marco Basaldella♥∗, Nigel Collier♣ ♣Language Technology Lab, TAL, University of Cambridge ♦Department of Data Science & AI, Monash University ♥Amazon Alexa ♣{fl399, zm324, nhc30}@cam.ac.uk ♦ehsan.shareghi@monash.edu ♥mbbasald@amazon.co.uk Abstract Despite the widespread success of self- supervised learning via masked language mod- els (MLM), accurately capturing fine-grained semantic relationships in the biomedical do- main remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we pro- pose SAPBERT, a pretraining scheme that self- aligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with 4M+ concepts. In contrast with previous pipeline- based hybrid systems, SAPBERT offers an el- egant one-model-for-all solution to the prob- lem of medical entity linking (MEL), achiev- ing a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific do- main, we achieve SOTA even without task- specific supervision. With substantial improve- ment over various domain-specific pretrained MLMs such as BIOBERT, SCIBERT and PUB- MEDBERT, our pretraining scheme proves to be both effective and robust.1 1 Introduction Biomedical entity2 representation is the founda- tion for a plethora of text mining systems in the medical domain, facilitating applications such as literature search (Lee et al., 2016), clinical decision making (Roberts et al., 2015) and relational knowl- edge discovery (e.g. chemical-disease, drug-drug and protein-protein relations, Wang et al. 2018). The heterogeneous naming of biomedical concepts ∗Work conducted prior to joining Amazon. 1For code and pretrained models, please visit: https: //github.com/cambridgeltl/sapbert. 2In this work, biomedical entity refers to the surface forms of biomedical concepts, which can be a single word (e.g. fever), a compound (e.g. sars-cov-2) or a short phrase (e.g. abnormal retinal vascular development). PUBMEDBERT + SAPBERT PUBMEDBERT Figure 1: The t-SNE (Maaten and Hinton, 2008) vi- sualisation of UMLS entities under PUBMEDBERT (BERT pretrained on PubMed papers) & PUBMED- BERT+SAPBERT (PUBMEDBERT further pretrained on UMLS synonyms). The biomedical names of differ- ent concepts are hard to separate in the heterogeneous embedding space (left). After the self-alignment pre- training, the same concept’s entity names are drawn closer to form compact clusters (right). poses a major challenge to representation learning. For instance, the medication Hydroxychloroquine is often referred to as Oxichlorochine (alternative name), HCQ (in social media) and Plaquenil (brand name). MEL addresses this problem by framing it as a task of mapping entity mentions to unified con- cepts in a medical knowledge graph.3 The main bottleneck of MEL is the quality of the entity rep- resentations (Basaldella et al., 2020). Prior works in this domain have adopted very sophisticated text pre-processing heuristics (D’Souza and Ng, 2015; Kim et al., 2019; Ji et al., 2020; …

Content truncated to 500 words