COMETA: A Corpus for Medical Entity Linking in the Social Media

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3122–3137, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3122 COMETA: A Corpus for Medical Entity Linking in the Social Media Marco Basaldella†,∗, Fangyu Liu†,∗, Ehsan Shareghi†,‡, Nigel Collier† †Language Technology Lab, University of Cambridge ‡ University College London †{mb2313, fl399, nhc30}@cam.ac.uk ‡e.shareghi@ucl.ac.uk Abstract Whilst there has been growing progress in En- tity Linking (EL) for general language, exist- ing datasets fail to address the complex nature of health terminology in layman’s language. Meanwhile, there is a growing need for appli- cations that can understand the public’s voice in the health domain. To address this we intro- duce a new corpus called COMETA, consist- ing of 20k English biomedical entity mentions from Reddit expert-annotated with links to SNOMED CT, a widely-used medical knowl- edge graph. Our corpus satisfies a combination of desirable properties, from scale and cover- age to diversity and quality, that to the best of our knowledge has not been met by any of the existing resources in the field. Through bench- mark experiments on 20 EL baselines from string- to neural-based models we shed light on the ability of these systems to perform com- plex inference on entities and concepts under 2 challenging evaluation scenarios. Our exper- imental results on COMETA illustrate that no golden bullet exists and even the best main- stream techniques still have a significant per- formance gap to fill, while the best solution re- lies on combining different views of data. 1 Introduction Social media has become a dominant means for users to share their opinions, emotions and daily experience of life. A large body of work has shown that informal exchanges such as online forums can be leveraged to supplement traditional approaches to a broad range of public health questions such as monitoring suicidal risk and depression (Benton et al., 2017b), domestic abuse (Schrading et al., 2015), cancer (Nzali et al., 2017), and epidemics (Aramaki et al., 2011; Joshi et al., 2019). One of the widely exercised steps to establish a semantic understanding of social media is En- ∗Equal contribution. diagnosed with gad where my benzos at ? Benzodiazepine SCTID: 372664007 Generalised anxiety disorder SCTID: 21897009 Glutamate decarboxylase SCTID: 41465008 Anxiety disorder Enzyme ? Is a Mention of Sedative (a) went to get bloods done at 11 30 am Blood test SCTID: 396550006 Blood SCTID: 87612001 Procedure Substance ? Is a Mention of (b) Figure 1: Examples of the EL inference challenges for user generated text in the health domain. tity Linking (EL), i.e., the task of linking entities within a text to a suitable concept in a reference Knowledge Graph (KG) (Liu et al., 2013; Yang and Chang, 2015; Yang et al., 2016; Ran et al., 2018). However, it is well-documented that poorly com- posed contexts, the ubiquitous presence of colloqui- alisms, shortened forms, typing/spelling mistakes, and out-of-vocabulary words introduce challenges for effective utilisation of social media text (Bald- win et al., 2013; Michel and Neubig, 2018). These …

Content truncated to 500 words