SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2216–2226 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data Maël Jullien1, Marco Valentino2, 1, Hannah Frost1, 3, Paul O’Regan3, Donal Landers3, André Freitas1, 2 Department of Computer Science, University of Manchester, United Kingdom1 Idiap Research Institute, Switzerland2 Digital Experimental Cancer Medicine Team, Cancer Research UK Manchester Institute3 {firstname.surname}@manchester.ac.uk Abstract This paper describes the results of Se- mEval 2023 task 7 – Multi-Evidence Natu- ral Language Inference for Clinical Trial Data (NLI4CT) – consisting of 2 tasks, a Natural Language Inference (NLI) task, and an evi- dence selection task on clinical trial data. The proposed challenges require multi-hop biomed- ical and numerical reasoning, which are of sig- nificant importance to the development of sys- tems capable of large-scale interpretation and retrieval of medical evidence, to provide per- sonalized evidence-based care. Task 1, the entailment task, received 643 sub- missions from 40 participants, and Task 2, the evidence selection task, received 364 submis- sions from 23 participants. The tasks are chal- lenging, with the majority of submitted systems failing to significantly outperform the majority class baseline on the entailment task, and we observe significantly better performance on the evidence selection task than on the entailment task. Increasing the number of model parame- ters leads to a direct increase in performance, far more significant than the effect of biomed- ical pre-training. Future works could explore the limitations of large models for generaliza- tion and numerical inference, and investigate methods to augment clinical datasets to allow for more rigorous testing and to facilitate fine- tuning. We envisage that the dataset, models, and re- sults of this task will be useful to the biomedical NLI and evidence retrieval communities. The dataset1, competition leaderboard2, and web- site3 are publicly available. 1 Introduction Clinical trials are indispensable for experimen- tal medicine as they test the efficacy and safety 1https://github.com/ai-systems/nli4ct 2https://codalab.lisn.upsaclay.fr/ competitions/8937#learn_the_details 3https://sites.google.com/view/nli4ct/ Figure 1: We propose two tasks for reasoning on clinical trial data expressed in natural language. Firstly, to pre- dict the entailment of a Statement and a CTR premise, and secondly, to extract evidence to support the label. of novel treatments (Avis et al., 2006). Clinical Trial Reports (CTRs) are documents that detail the methodology and results of a trial, implemented to guide personalized and targeted interventions for patients. However, there are 400,000+ published CTRs, with an increasing number being published every year (Bastian et al., 2010), making it imprac- tical to manually carry out comprehensive evalua- tions of all the relevant literature when designing new treatment protocols (DeYoung et al., 2020). To address this challenge, Natural Language In- ference (NLI) (Bowman et al., 2015; Devlin et al., 2019) offers a potential solution for the large-scale interpretation and retrieval of medical evidence, to support a higher level of precision and efficiency in personalized evidence-based care (Sutton et al., 2020). SemEval-2023 Task 7 – Multi-Evidence Nat- ural Language Inference for Clinical Trial Data …

Content truncated to 500 words