FMI-SU at SemEval-2023 Task 7: Two-level Entailment Classification of Clinical Trials Enhanced by Contextual Data Augmentation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1454–1462 July 13-14, 2023 ©2023 Association for Computational Linguistics FMI-SU at SemEval-2023 Task 7: Two-level Entailment Classification of Clinical Trials Enhanced by Contextual Data Augmentation Sylvia Vassileva FMI, Sofia University svasileva@fmi.uni-sofia.bg Georgi Grazhdanski FMI, Sofia University ggrazhdans@uni-sofia.bg Svetla Boytcheva Ontotext & FMI, Sofia University svetla.boytcheva@ontotext.com svetla@uni-sofia.bg Ivan Koytchev FMI, Sofia University koychev@fmi.uni-sofia.bg Abstract The paper presents an approach for solving Se- mEval 2023 Task 7 - identifying the inference relation in a clinical trials dataset. The system has two levels for retrieving relevant clinical trial evidence for a statement and then classify- ing the inference relation based on the relevant sentences. In the first level, the system clas- sifies the evidence-statement pairs as relevant or not using a BERT-based classifier and con- textual data augmentation (subtask 2). Using the relevant parts of the clinical trial from the first level, the system uses an additional BERT- based classifier to determine whether the rela- tion is entailment or contradiction (subtask 1). In both levels, the contextual data augmenta- tion is showing a significant improvement in the F1 score on the test set of 3.7% for subtask 2 and 7.6% for subtask 1, achieving final F1 scores of 82.7% for subtask 2 and 64.4% for subtask 1. 1 Introduction A huge amount of clinical trial reports are gen- erated annually, and the ability to automatically extract and analyze data from these reports can help healthcare professionals stay abreast of the lat- est trends and findings from these trials. SemEval 2023 Task 7: Multi-evidence Natural Language In- ference for Clinical Trial Data consists of a dataset of breast cancer clinical trial reports and related statements in English (Jullien et al., 2023). The goal is to determine the trial-statement inference relationship (subtask 1) and to extract supporting sentences from the trial text (subtask 2). The proposed system reverses the task order and redefines subtask 2 as identifying the sentences from the clinical trial text relevant to the statement, similar to Question-Answering Natural Language Inference (QNLI) (Wang et al., 2018). After finding these sentences, they are used as input for classi- fying the inference relation - entailment or contra- diction of the statement with respect to the premise (subtask 1). We evaluated various transformer mod- els based on the original BERT architecture (Devlin et al., 2018) and pre-trained on biomedical/clinical data and a contextual data augmentation approach that improves the model’s ability to discriminate related sentences and classify them correctly. The system showed 82.7% F1 score on the test set and ranked fifth in the leaderboard for subtask 2, 2.6% behind the first-place model. Subtask 1 is more challenging and the system showed 64.4% F1 score on the test set there. The system struggles the most when the statement requires performing quan- titative reasoning to determine the relationship. The code related to this task is available on GitHub 1. 2 Background The task uses a dataset of breast cancer clinical trial reports, statements, inference …

Content truncated to 500 words