UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1744–1753 June 6–11, 2021. ©2021 Association for Computational Linguistics 1744 UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, Alexander Wong University of Waterloo, Waterloo, Canada {gmichalo, yuanxin.wang, hussam.kaka, helen.chen, alexander.wong}@uwaterloo.ca Abstract Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific corpora. However, such models do not take into consideration structured expert domain knowledge from a knowledge base. We introduce UmlsBERT, a contextual em- bedding model that integrates domain knowl- edge during the pre-training process via a novel knowledge augmentation strategy. More specifically, the augmentation on UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus is performed in two ways: (i) connecting words that have the same underlying ‘concept’ in UMLS and (ii) lever- aging semantic type knowledge in UMLS to create clinically meaningful input embed- dings. By applying these two strategies, Umls- BERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models on common named- entity recognition (NER) and clinical natural language inference tasks. 1 Introduction In recent years, the volume of data being collected in healthcare has grown considerably. A signifi- cant proportion of the data is in text form, which requires advanced Natural Language Processing (NLP) models to process. This has led to the cre- ation of high-performing, optimized NLP models focused on the biomedical domain. Contextual word embedding models, such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) have achieved state-of-the-art results in many NLP tasks. Initially tested in a general domain, these models have also been successfully applied in the biomedical domain by pre-training them on biomedical corpora, leading to the best perfor- mances in a variety of biomedical NLP tasks (Lee et al., 2019), (Alsentzer et al., 2019). However, cur- rent biomedical applications of transformer-based Natural Language Understanding (NLU) models do not incorporate structured expert domain knowl- edge from a knowledge base into their embedding pre-training process. The Unified Medical Language System (UMLS) (Bodenreider, 2004) Metathesaurus is a com- pendium of many biomedical terminologies with the associated information, such as synonyms and categorical groupings. It allows for the connec- tion of words that represent the same or similar ‘concept’. For example, the words ‘lungs’ and ‘pul- monary’ share a similar meaning and thus can be mapped to the same concept unique identifier (CUI) CUI: C0024109. Additionally, UMLS allows the grouping of concepts according to their semantic type (McCray et al., 2001). For example, ‘skeleton’ and ‘skin’ have the same ‘Body System’ seman- tic type, and ‘inflammation’ and ‘bleed’ are in the ‘Pathologic Function’ semantic type. In this paper, we present and publicly release1 a novel architecture for augmenting contextual em- beddings with clinical domain knowledge. Specif- ically: (i) We are the first, to the best …

Content truncated to 500 words