Stanford MLab at SemEval 2023 Task 7: Neural Methods for Clinical Trial Report NLI

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1769–1775 July 13-14, 2023 ©2023 Association for Computational Linguistics Stanford MLab at SemEval 2023 Task 7: Neural Methods for Clinical Trial Report NLI Conner Takehana, Dylan Lim, Emirhan Kurtulus, Ramya Iyer, Ellie Tanimura, Pankhuri Aggarwal, Molly Cantillon, Alfred Yu, Sarosh Khan, Nathan Chi∗, and Ryan A. Chi∗ Stanford University {conner7, dylanlim, emirhank, ramya.iyer, etanim, pankhuri, mcan, ayu1001, skhan44, nchi1, ryanchi} @stanford.edu Abstract We present a system for natural language in- ference in breast cancer clinical trial reports, as framed by SemEval 2023 Task 7: Multi- evidence Natural Language Inference for Clini- cal Trial Data. In particular, we propose a suite of techniques for two related inference sub- tasks: entailment and evidence retrieval. The purpose of the textual entailment identification subtask is to determine the inference relation (either entailment or contradiction) between given statement pairs, while the goal of the ev- idence retrieval task is to identify a set of sen- tences that support this inference relation. To this end, we propose fine-tuning Bio+Clinical BERT, a BERT-based model pre-trained on clin- ical data. Along with presenting our system, we analyze our architectural decisions in the context of our model’s accuracy and conduct an error analysis. Overall, our system ranked 20 / 40 on the entailment subtask. 1 Introduction Clinical trial reports (CTRs) detail the treatments and reactions that patients undergo over the course of a clinical study. The growing corpus of CTRs, coupled with a lack of adequate analytical tools, makes it increasingly difficult for clinical practition- ers to provide up-to-date, personalized, evidence- based care (DeYoung et al., 2020). Improvements in the reliability of CTRs have not been accompa- nied by developments in the ability to analyze and compare them, leading to a substantial disparity be- tween the quality of CTRs and physicians’ abilities to capitalize on them. (Butcher et al., 2019). While natural language inference (NLI) mod- els demonstrate impressive capabilities in inferring logical relationships between texts, applications in the medical domain come with a unique set of chal- lenges (Percha et al., 2021). One issue is that ac- curately inferring relationships between entities in ∗Co-senior authors. clinical data requires knowledge of domain-specific jargon (Sushil et al., 2021). Additionally, because scientific communication can be highly variable in its technical language, abbreviations, and format- ting, successful NLI systems must be trained on a large representative corpus of varied data. Finally, it can be difficult for NLI systems to distinguish between non-entailed subsequences like "John be- lieves Martha is sick" and "John believes Martha" (McCoy and Linzen, 2018). The precise interpretation and extraction of ev- idence from CTRs could drastically diminish the current disconnect between research findings and clinical practices, as it will be much easier for clin- icians to parse new publications and stay updated on the most effective treatment practices (?). In this paper, we experiment with a variety of mod- ern and traditional machine learning approaches to accomplish this task. Ultimately, we find that Bio+Clinical BERT and …

Content truncated to 500 words