Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3457 - 3463 July 10-15, 2022 ©2022 Association for Computational Linguistics Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation Luyao Shi Tanveer F. Syeda-Mahmood Tyler Baldwin IBM Research, Almaden Research Center, San Jose, CA, USA luyao.shi@ibm.com {stf, tbaldwin}@us.ibm.com Abstract Many clinical informatics tasks that are based on electronic health records (EHR) need rel- evant patient cohorts to be selected based on findings, symptoms and diseases. Frequently, these conditions are described in radiology re- ports which can be retrieved using information retrieval (IR) methods. The latest of these tech- niques utilize neural IR models such as BERT trained on clinical text. However, these meth- ods still lack semantic understanding of the un- derlying clinical conditions as well as ruled out findings, resulting in poor precision dur- ing retrieval. In this paper we combine clinical finding detection with supervised query match learning. Specifically, we use lexicon-driven concept detection to detect relevant findings in sentences. These findings are used as queries to train a Sentence-BERT (SBERT) model us- ing triplet loss on matched and unmatched query-sentence pairs. We show that the pro- posed supervised training task remarkably im- proves the retrieval performance of SBERT. The trained model generalizes well to unseen queries and reports from different collections. 1 Introduction Electronic health record (EHR) retrieval is impor- tant for clinicians, staff and researchers. The tools for performing clinically relevant searches could aid in many use cases such as clinical decision sup- port (Syeda-Mahmood, 2010), auditing, revenue cycle management, and cohort selection for clin- ical studies. Frequently, these searches involve retrieval of patients based on clinical findings that are often captured in unstructured textual reports such as radiology reports, encounter notes, etc. Un- like structured query-based lookup of EHR, re- trieval of unstructured (free-text) EHRs is much more challenging, requiring a semantic understand- ing of the underlying clinical conditions present or absent. Conventional exact or approximate term- based retrieval methods such as BM25 (Robertson and Zaragoza, 2009) often perform poorly in re- sponse to ad-hoc queries (Chamberlin et al., 2020), as these methods lack the ability of semantic under- standing of the clinical as well as language context. With the emergence of deep learning encoding mod- els, new retrieval methods have emerged with stud- ies showing BERT-based neural methods outper- forming BM25 on multiple retrieval benchmarks (Yilmaz et al., 2019a; Chang et al., 2020; Nogueira and Cho, 2019; Yilmaz et al., 2019b; Qiao et al., 2019). The BERT-based retrieval methods can be classified into two categories: the cross-attention (or interaction-based) models (Yilmaz et al., 2019a; Nogueira and Cho, 2019; Yilmaz et al., 2019b) and the embedding-based (or representation-based) models (Chang et al., 2020; Reimers and Gurevych, 2019). While the BERT-style cross-attention mod- els are very successful, they cannot be directly applied to large-scale retrieval problems because computing the similarity score for every possible query-document pair during inference can be pro- hibitively expensive. …

Content truncated to 500 words