Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1863–1870 July 13-14, 2023 ©2023 Association for Computational Linguistics Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports Juraj Vladika and Florian Matthes Department of Computer Science, Technical University of Munich {juraj.vladika, matthes}@tum.de Abstract With the increasing number of clinical trial re- ports generated every day, it is becoming hard to keep up with novel discoveries that inform evidence-based healthcare recommendations. To help automate this process and assist med- ical experts, NLP solutions are being devel- oped. This motivated the SemEval-2023 Task 7, where the goal was to develop an NLP system for two tasks: evidence retrieval and natural lan- guage inference from clinical trial data. In this paper, we describe our two developed systems. The first one is a pipeline system that models the two tasks separately, while the second one is a joint system that learns the two tasks simul- taneously with a shared representation and a multi-task learning approach. The final system combines their outputs in an ensemble system. We formalize the models, present their charac- teristics and challenges, and provide an analysis of achieved results. Our system ranked 3rd out of 40 participants with a final submission. 1 Introduction Clinical trials are research studies carried out in hu- man subjects to assess the effectiveness and safety of medical, surgical, or behavioral interventions (Friedman et al., 2015). These investigations con- stitute the main approach that medical researchers use to determine whether a novel treatment such as a new medication, procedure, diet, or medical device is safe and efficient in humans. Clinical trials are often used to compare the effectiveness of a new treatment against the standard treatment or placebo treatment and to assess its adverse ef- fects. When performed rigorously, clinical trials present the most valuable resource for informing evidence-based healthcare decisions. On average, more than 100 reports of CTs are published every day (Zarin et al., 2019). Keeping up with all their results and novel discoveries is a time-consuming and often practically impossi- ble endeavor. This has brought to light dedicated Figure 1: The task consists of predicting whether a given claim entails or contradicts the clinical trial report based on the evidence found in it. organizations, such as Cochrane,1 that manually synthesize these clinical outcomes (Higgins et al., 2019). Nevertheless, they struggle to keep up with the ever-increasing amount of literature and new studies. For this purpose, automated approaches based on Machine Learning (ML) and Natural Lan- guage Processing (NLP) can be developed to facil- itate the process of inferring new knowledge and finding evidence from clinical trial reports. The SemEval-2023 Task 7, titled Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT), focused on developing NLP systems for making conclusions and finding evidence in clin- ical trial reports (CTRs) (Jullien et al., 2023). It featured two subtasks that were strongly coupled together. The first subtask was, given a clinical trial document and …

Content truncated to 500 words