I2R at SemEval-2023 Task 7: Explanations-driven Ensemble Approach for Natural Language Inference over Clinical Trial Data

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1630–1635 July 13-14, 2023 ©2023 Association for Computational Linguistics Abstract In this paper, we describe our system for SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. Given a CTR premise, and a statement, this task involves 2 sub-tasks (i) identifying the inference relation between CTR - statement pairs (Task 1: Textual Entailment), and (ii) extracting a set of supporting facts, from the premise, to justify the label predicted in Task 1 (Task 2: Evidence Retrieval). We adopt an explanation driven NLI approach to tackle the tasks. Given a statement to verify, the idea is to first identify relevant evidence from the target CTR(s), perform evidence level inferences and then ensemble them to arrive at the final inference. We have experimented with various BERT based models and T5 models. Our final model uses T5 base that achieved better performance compared to BERT models. In summary, our system achieves F1 score of 70.1% for Task 1 and 80.2% for Task 2. We ranked 8th respectively under both the tasks. Moreover, ours was one of the 5 systems that ranked within the Top 10 under both tasks. 1 Introduction Natural language inference (NLI) is the task of detecting inferential relationships between a premise text and a hypothesis text (MacCartney and Manning, 2009), which is considered fundamental in natural language understanding (NLU) research (Bowman et al., 2015). The objective is to determine whether hypothesis h is true (‘entailment’), false (‘contradiction’), or un- determined (‘neutral’) given the premise P. This task, formerly known as recognizing textual entailment (RTE) (Dagan et al., 2005) has long been a popular task among researchers. Moreover, contribution of datasets from past shared tasks (Dagan et al., 2009), and recent research (Bowman et al., 2015; Williams et al., 2018) have pushed the boundaries for this seemingly simple, but challenging problem. NLI brings an opportunity to support the large- scale interpretation and retrieval of medical evidence. Understanding the contextual evidence will support personalized care for patients (Sutton et al.,2020), e.g., analysis of Clinical Trial Reports (CTRs). This is especially useful as, in the past few years, the number of publications of CTRs has increased exponentially and it has become impracticable for clinical practitioners to stay updated (DeYoung et al., 2020). SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (Jullien et al., 2023) attempts to capture and investigate this opportunity via two shared tasks namely (i) Task 1: Textual Entailment: Given a statement, which make some type of claim about the information contained in one of the sections in the CTR premise, identify the inference relation (either entailment or contradiction) between CTR - statement pairs. (ii) Task 2: Given a CTR premise, and a statement, extract a set of supporting facts, from the premise, to justify the label predicted in Task 1. Several NLI systems have been proposed in the literature. (Bowman et al., 2015, Romanov et al., 2015, Marelli et al., 2014, Khot et …

Content truncated to 500 words