THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrieval

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1681–1690 July 13-14, 2023 ©2023 Association for Computational Linguistics THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrieval Yuxuan Zhou1∗, Ziyu Jin2,3∗, Meiwei Li2,3, Miao Li1, Xien Liu1, Xinxin You2,3, Ji Wu1 1Department of Electronic Engineering, Tsinghua University, Beijing 100084, China 2THiFLY Research, Tsinghua University, Beijing 100084, China 3State Key Laboratory of Cognitive Intelligence, Hefei, Anhui 230088, China Abstract The NLI4CT task aims to entail hypotheses based on Clinical Trial Reports (CTRs) and retrieve the corresponding evidence support- ing the justification. This task poses a sig- nificant challenge, as verifying hypotheses in the NLI4CT task requires the integration of multiple pieces of evidence from one or two CTR(s) and the application of diverse levels of reasoning, including textual and numeri- cal. To address these problems, we present a multi-granularity system for CTR-based tex- tual entailment and evidence retrieval in this paper. Specifically, we construct a Multi- granularity Inference Network (MGNet) that exploits sentence-level and token-level encod- ing to handle both textual entailment and evi- dence retrieval tasks. Moreover, we enhance the numerical inference capability of the sys- tem by leveraging a T5-based model, SciFive, which is pre-trained on the medical corpus. Model ensembling and a joint inference method are further utilized in the system to increase the stability and consistency of inference. The system achieves f1-scores of 0.856 and 0.853 on textual entailment and evidence retrieval tasks, resulting in the best performance on both subtasks. The experimental results corrobo- rate the effectiveness of our proposed method. Our code is publicly available at https:// github.com/THUMLP/NLI4CT. 1 Introduction In recent years, with the fast development of digital health, there has been a surge in the publication of Clinical Trial Reports (CTRs). Currently, there are over 10,000 CTRs for Breast Cancer alone1. This proliferation of CTRs has enabled the construc- tion of a Natural Language Inference (NLI) system that can aid medical interpretation and evidence retrieval for personalized evidence-based care. The *: equal contribution 1https://sites.google.com/view/nli4ct/ home Figure 1: An Example of NLI4CT. NLI4CT task (Jullien et al., 2023) focuses on con- structing an explainable multi-evidence NLI sys- tem based on CTRs for Breast Cancer. It consists of two sub-tasks: textual entailment (Task A) and evidence retrieval (Task B). Task A requires the system to determine if the CTRs entail or contra- dict the given hypotheses. Task B requires the system to find evidence in the CTRs necessary to justify the prediction in Task A. The primary difficulty inherent in NLI4CT is that the process of hypothesis verification frequently involves the integration of several pieces of evidence from the premise. In certain cases, justifying a hypothesis re- quires a comparison of two separate premise CTRs. Moreover, unlike other clinical NLI benchmarks (e.g., MedNLI), NLI4CT presents a unique feature containing four distinct types of premises - Inter- vention, Eligibility, Results, and Adverse Events - that align with the four sections typically found in each CTR. Validating …

Content truncated to 500 words