Attention Networks for Augmenting Clinical Text with Support Sets for Diagnosis Prediction

Proceedings of the 29th International Conference on Computational Linguistics, pages 4765–4775 October 12–17, 2022. 4765 Attention Networks for Augmenting Clinical Text with Support Sets for Diagnosis Prediction Paul Grundmann∗, Tom Oberhauser∗, Felix Gers and Alexander Löser Berliner Hochschule für Technik Luxemburger Str. 10, 13353 Berlin {pgrundmann, toberhauser, gers, aloeser}@bht-berlin.de ∗Both authors contributed equally. Abstract Diagnosis prediction on admission notes is a core clinical task. However, these notes may in- completely describe the patient. Also, clinical language models may suffer from idiosyncratic language or imbalanced vocabulary for describ- ing diseases or symptoms. We tackle the task of diagnosis prediction, which consists of pre- dicting future patient diagnoses from clinical texts at the time of admission. We improve the performance on this task by introducing an additional signal from support sets of diag- nostic codes from prior admissions or as they emerge during differential diagnosis. To en- hance the robustness of diagnosis prediction methods, we propose to augment clinical text with potentially complementary set data from diagnosis codes from previous patient visits or from codes that emerge from the current admission as they become available through diagnostics. We discuss novel attention net- work architectures and augmentation strategies to solve this problem. Our experiments reveal that support sets improve the performance dras- tically to predict less common diagnosis codes. Our approach clearly outperforms the previous state-of-the-art PubMedBERT baseline by up 3% points. Furthermore, we find that support sets drastically improve the performance for pregnancy- and gynecology-related diagnoses up to 32.9 % points compared to the baseline. 1 Introduction Pre-trained large language models such as Clini- calBERT (Alsentzer et al., 2019) or PubMedBERT (Gu et al., 2021) are commonly used in the medical domain to predict diagnoses from admission notes (Hashir and Sawhney, 2020; Sushil et al., 2018a; van Aken et al., 2021). Admission and discharge notes are valuable information sources about doc- tors’ decisions about patients and the outcomes. However, the vocabulary in these notes is often insufficient to describe the patients’ clinical pheno- type fully. Also, clinical text frequently contains idiosyncratic vocabulary, uncommon abbreviations and differs from clinic to clinic in writing style. Moreover, evidence for clinical diseases in the text is imbalanced. In particular for less common or even rare diseases (see also Figure 1) not much text evidence exists. Finally, pre-trained language mod- els can suffer from limited access to training data because of silos or data-privacy concerns. These factors can lead to poor performance in predicting outcomes on clinical text with pre-trained language models. Multimodal Patient Representation. Miotto et al. (2016) and Topol (2019) therefore propose to augment text with potential complementary multi- modal data into a reusable deep patient represen- tation to improve clinical prediction tasks. For example, recent work surveys to augment text with image data (Esteva et al., 2021), with complemen- tary medical text books (van Aken et al., 2021), ontologies (Cai et al., 2020) or time series data (Yang and Wu, 2021). Improving predictions with text and set data. A particularly powerful source to augment …

Content truncated to 500 words