ITTC at SemEval 2023-Task 7: Document Retrieval and Sentence Similarity for Evidence Retrieval in Clinical Trial Data

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2338–2342 July 13-14, 2023 ©2023 Association for Computational Linguistics ITTC at SemEval 2023-Task 7: Document Retrieval and Sentence Similarity for Evidence Retrieval in Clinical Trial Data Rahmad Mahendra and Damiano Spina and Karin Verspoor RMIT University Melbourne, Australia rahmad.mahendra@student.rmit.edu.au {damiano.spina,karin.verspoor}@rmit.edu.au Abstract This paper describes the submissions of the Nat- ural Language Processing (NLP) team from the Australian Research Council Industrial Trans- formation Training Centre (ITTC) for Cogni- tive Computing in Medical Technologies to the SemEval 2023 Task 7, i.e., multi-evidence nat- ural language inference for clinical trial data (NLI4CT). More specifically, we were working on subtask 2 whose objective is to identify the relevant parts of the premise from clinical trial report that justify the truth of information in the statement. We approach the evidence retrieval problem as a document retrieval and sentence similarity task. Our results show that the task poses some challenges which involve dealing with complex sentences and implicit evidences. 1 Introduction Clinical Trial Reports (CTRs) are an important source of evidence for clinical practitioners, which provide information on the safety and efficacy of new treatments and interventions (Akobeng, 2005). However, in recent years, the number of CTRs published has increased significantly, making it dif- ficult for clinicians to stay up-to-date with the latest research (Bastian et al., 2010). This challenge is further exacerbated by the growing number of pa- tient cases that require customized treatment plans. As the result, it has become infeasible for prac- titioners to manually sift through all the current literature to provide personalized evidence-based care to their patients. To overcome this issue, there is a need for tech- nologies that can help clinicians efficiently review and interpret the latest medical literature. One such tool is Natural Language Inference (NLI), a task in natural language processing (NLP) to determine the semantic relation between two sentences (Bow- man et al., 2015). NLI-powered systems offer a promising solution by extracting relevant informa- tion from CTRs and enabling automated evidence synthesis (DeYoung et al., 2020). By leveraging NLI, we could significantly improve the way we link the most current evidence to offer individual- ized evidence-based care (Sutton et al., 2020). Task. SemEval 2023 Task 7: Multi-evidence Nat- ural Language Inference for Clinical Trial Data (NLI4CT) consists of two subtasks, i.e., textual entailment and evidence retrieval (Jullien et al., 2023). In the first subtask, the system is challenged to decide whether a statement making claims about a single CTR (or comparing 2 CTRs) is true or not. In other words, the objective of Subtask 1 is to determine the inference relation (entailment vs. contradiction) between CTR and statement pairs. Subtask 2 involves a CTR premise and statement, and requires the system to select relevant facts ex- tracted from the premise to support the statement. These facts are necessary to justify the inference label of Subtask 1. Dataset. The NLI4CT shared task uses a collec- tion of breast cancer CTRs. A team of clinicians generated a …

Content truncated to 500 words