Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8754–8769 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8754 Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case Jingqing Zhang1,2, Luis Bolanos2, Tong Li2, Ashwani Tanwar2, Guilherme Freire2 Xian Yang3, Julia Ive1, Vibhor Gupta2, Yike Guo1,2,3 1Data Science Institute, Imperial College London, UK 2Pangaea Data Limited, UK, USA 3Hong Kong Baptist University, Hong Kong SAR, China jzhang,lbolanos,tli,atanwar,gfreire,vgupta@pangaeadata.ai j.ive,y.guo@imperial.ac.uk xianyang@comp.hkbu.edu.hk Abstract Contextualised word embeddings is a pow- erful tool to detect contextual synonyms. However, most of the current state-of-the-art (SOTA) deep learning concept extraction meth- ods remain supervised and underexploit the po- tential of the context. In this paper, we propose a self-supervised pre-training approach which is able to detect contextual synonyms of con- cepts being training on the data created by shal- low matching. We apply our methodology in the sparse multi-class setting (over 15,000 con- cepts) to extract phenotype information from electronic health records. We further investi- gate data augmentation techniques to address the problem of the class sparsity. Our ap- proach achieves a new SOTA for the unsuper- vised phenotype concept annotation on clinical text on F1 and Recall outperforming the pre- vious SOTA with a gain of up to 4.5 and 4.0 absolute points, respectively. After fine-tuning with as little as 20% of the labelled data, we also outperform BioBERT and ClinicalBERT. The extrinsic evaluation on three ICU bench- marks also shows the benefit of using the phe- notypes annotated by our model as features. 1 Introduction Supervised fine-tuning on the top of the BERT- based models has recently become the standard approach in Information Extraction delivering state- of-the-art (SOTA) results across different tasks (De- vlin et al., 2019). The dependence of these models on the availability of the costly human-annotations remains a serious obstacle towards a large scale deployment of such models. This problem is es- pecially actual in the clinical domain with limited availability of experts. In the self-supervised setting, automatic anno- tations are cheap to produce. Some rule-based automatic labelers, such as CheXpert (Irvin et al., 2019), which is built on NegBio (Peng et al., 2018), are often used to create training data for supervised BERT-based models (e.g., automatic annotators of radiology reports (Smit et al., 2020)). Those mod- els usually generalise over a small set of classes (under 20). Other automatic labelers exploit ontologies. For example, the UMLS (Unified Medical Language System) ontology (Bodenreider, 2004b) is almost predominantly used to match linguistics patterns in clinical text to medical concepts (e.g., using the MetaMap tool (Aronson, 2006)). Due to the complexity of the Information Extraction task in this challenging setting (sparse multi-class), the ap- proaches that use the data annotated (e.g., (Arbabi et al., 2019; Kraljevic et al., 2019; Tiwari et al., 2020)) mostly rely on non-contextualised embed- dings focusing on the detection precision. However, especially for clinical text, which is noisier and ex- hibits a variety of clinical expressions requiring disambiguation, relying on the context …

Content truncated to 500 words