MDC at SemEval-2023 Task 7: Fine-tuning Transformers for Textual Entailment Prediction and Evidence Retrieval in Clinical Trials

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 125–129 July 13-14, 2023 ©2023 Association for Computational Linguistics Bf3R at SemEval-2023 Task 7: a text similarity model for textual entailment and evidence retrieval in clinical trials and animal studies Mariana Neves German Centre for the Protection of Laboratory Animals (Bf3R), German Federal Institute for Risk Assessment (BfR), Berlin, Germany Abstract We describe our participation on the Multi- evidence Natural Language Inference for Clini- cal Trial Data (NLI4CT) of SemEval’23. The organizers provided a collection of clinical tri- als as training data and a set of statements, which can be related to either a single trial or to a comparison of two trials. The task con- sisted of two sub-tasks: (i) textual entailment (Task 1) for predicting whether the statement is supported (Entailment) or not (Contradiction) by the corresponding trial(s); and (ii) evidence retrieval (Task 2) for selecting the evidences (sentences in the trials) that support the deci- sion made for Task 1. We built a model based on a sentence-based BERT similarity model which was pre-trained on ClinicalBERT em- beddings. Our best results on the official test sets were f-scores of 0.64 and 0.67 for Tasks 1 and 2, respectively. 1 Introduction The increase on the number of digital resources in biomedicine requires more adequate text mining methods to allow experts to stay up-to-date with the recent findings (DeYoung et al., 2020). Task 7 of the SemEval’23 (Jullien et al., 2023), namely, Multi-evidence Natural Language Infer- ence for Clinical Trial Data (NLI4CT)1, aimed at automatic processing clinical trials for two particu- lar sub-tasks. In the scope of the shared task, the organizers released a collection of clinical trials, each of them composed of four sections, namely, “Intervention”, “Eligibility”, “Results”, and “Ad- verse Events”. Each of these sections contains one or more sentences. The shared task was based on a set of the so- called “Premise-Statement-Evidence” items, which could be related to one or two clinical trials. If re- lated to only one trial, the item was set as “Single”, otherwise it is a comparison between two trials, 1https://sites.google.com/view/nli4ct/ i.e., set as “Comparison”. Each Premise-Statement- Evidence item refers to only one particular section of the trial and contained a statement that made a claim about one or two trials, and which was the main input to be considered in the tasks. We present two examples in Table 1. Based on these items, two tasks were proposed: “Textual Entailment” (Task 1) and “Evidence retrieval” (Task 2). The “Text Entailment” task consisted of automat- ically classifying the relation between the statement and the trial. Two values of labels were possible (cf. Table 1): “Entailment”, i.e., the trial supports the claim, or “Contradiction”, i.e., the trial contra- dicts the claim. Both values are possible for either single or comparison items. Only one of the labels is allowed for each item. The “Evidence retrieval” task consisted of find- ing evidences that support the claim. The evidences can be one or more sentences …

Content truncated to 500 words