Aristoxenus at SemEval-2023 Task 4: A Domain-Adapted Ensemble Approach to the Identification of Human Values behind Arguments

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1–9 July 13-14, 2023 ©2023 Association for Computational Linguistics KnowComp at SemEval-2023 Task 7: Fine-tuning Pre-trained Language Models for Clinical Trial Entailment Identification Weiqi Wang∗, Baixuan Xu∗, Tianqing Fang, Lirong Zhang, Yangqiu Song Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China wwangbw@cse.ust.hk, bxuan@connect.ust.hk, tfangaa@cse.ust.hk lzhangdo@connect.ust.hk, yqsong@cse.ust.hk Abstract In this paper, we present our system for the tex- tual entailment identification task as a subtask of the SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. The entailment identification task aims to determine whether a medical statement af- firms a valid entailment given a clinical trial premise or forms a contradiction with it. Since the task is inherently a text classification task, we propose a system that performs binary clas- sification given a statement and its associated clinical trial. Our proposed system leverages a human-defined prompt to aggregate the in- formation contained in the statement, section name, and clinical trials. Pre-trained language models are then finetuned on the prompted in- put sentences to learn to discriminate the in- ferential relation between the statement and clinical trial. To validate our system, we con- duct extensive experiments with a wide vari- ety of pre-trained language models. Our best system is built on DeBERTa-v3-large, which achieves an F1 score of 0.764 and secures the fifth rank in the official leaderboard. Further analysis indicates that leveraging our designed prompt is effective, and our model suffers from a low recall. Our code and pre-trained mod- els are available at https://github.com/HKUST- KnowComp/NLI4CT. 1 Introduction Recently, the proliferation of Clinical Trial Re- ports (CTRs) provides a large-scale reference base of scientific and factual knowledge for medical practitioners to make evidence-based clinical di- agnoses (Zlabinger et al., 2020). However, it also posits the challenge that clinical practitioners can- not memorize all current literature in order to pro- vide up-to-date personalized evidence-based med- ical care (DeYoung et al., 2020). With the recent advances in Natural Language Processing (NLP) ∗ Equal Contribution Entailment Contradiction PRIMARY TRIAL ADVERSE EVENTS 1: • Total: 69/258 (26.74%) • Anaemia 3/258 (1.16%) • Febrile neutropenia 13/258 (5.04%) • … • … PRIMARY TRIAL ADVERSE EVENTS 2: • Total: 64/224 (28.57%) • Anaemia 2/224 (0.89%) • Febrile neutropenia 4/224 (1.79%) • … • … PRIMARY TRIAL INTERVENTION 1: • Letrozole, Breast Enhancement, Safety • Single arm of healthy postmenopausal women to have two breast MRI (baseline and post-treatment). Letrozole of 12.5 mg/day is given for three successive days just prior to the second MRI. SECONDARY TRIAL INTERVENTION 1: • Healthy Volunteers • Healthy women will be screened for Magnetic Resonance Imaging (MRI) contraindications, and … • Magnetic resonance imaging: Patients and healthy volunteers will be first screened for MRI contraindications. STATEMENT The primary trial and the secondary trial both used MRI for their interventions. STATEMENT More than 1/3 of patients in cohort 1 of the primary trial experienced an adverse event. Figure 1: A demonstration of textual entailment and contradiction …

Content truncated to 500 words