Biomedical Concept Relatedness – A large EHR-based benchmark

Proceedings of the 28th International Conference on Computational Linguistics, pages 6565–6575 Barcelona, Spain (Online), December 8-13, 2020 6565 Biomedical Concept Relatedness – A large EHR-based benchmark Claudia Schulz and Josh Levy-Kramer and Camille Van Assel and Miklos Kepes and Nils Hammerla Babylon Health London, SW3 3DD, UK Abstract A promising application of AI to healthcare is the retrieval of information from electronic health records (EHRs), e.g. to aid clinicians in finding relevant information for a consultation or to recruit suitable patients for a study. This requires search capabilities far beyond simple string matching, including the retrieval of concepts (diagnoses, symptoms, medications, etc.) related to the one in question. The suitability of AI methods for such applications is tested by predicting the relatedness of concepts with known relatedness scores. However, all existing biomedical concept relatedness datasets are notoriously small and consist of hand-picked concept pairs. We open- source a novel concept relatedness benchmark overcoming these issues: it is six times larger than existing datasets and concept pairs are chosen based on co-occurrence in EHRs, ensuring their relevance for the application of interest. We present an in-depth analysis of our new dataset and compare it to existing ones, highlighting that it is not only larger but also complements existing datasets in terms of the types of concepts included. Initial experiments with state-of-the-art embedding methods show that our dataset is a challenging new benchmark for testing concept relatedness models. 1 Introduction The adoption of electronic health records (EHRs) facilitates interoperability, meaning that more and more information from different sources is being stored about a patient. This makes it increasingly challenging for doctors to efficiently filter a patient’s record for relevant information during a consultation without missing anything. This is particularly problematic since consultations are time-constrained. In the UK for example, general practitioner (GP) doctors usually have less than 10 minutes to consult a patient (Flaxman, 2015; Salisbury, 2019). EHRs consist not only of free-text records but are furthermore tagged by doctors with medical concept codes. This coding is aimed at standardising health records to enable, e.g., the seamless transfer of patient information between practices and the analysis of health data from different practices (Sinha et al., 2012; Morrison et al., 2014). In addition, coded EHRs allow for the search of concept codes in an EHR. However, retrieving not only the exact concept in question but also related ones, as done by doctors when reading a patient’s record, is less straight-forward. For a patient with potential liver failure, related information of interest to a doctor in the patient’s history are for example ‘alcohol abuse’, as it is a risk factor of liver failure, and ‘jaundice’, a symptom of liver failure. Other risk factors, symptoms, treatments, conditions, or tests associated with liver failure would also be considered relevant. Concept representation models, such as embeddings or ontology-based methods (McInnes et al., 2009; Pivovarov and Elhadad, 2012; Henry et al., 2018; Smalheiser et al., 2019; Park et al., 2019), have been developed to tackle the task of …

Content truncated to 500 words