Clemson NLP at SemEval-2023 Task 7: Applying GatorTron to Multi-Evidence Clinical NLI

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1598–1602 July 13-14, 2023 ©2023 Association for Computational Linguistics Clemson NLP at SemEval-2023 Task 7: Applying GatorTron to Multi-Evidence Clinical NLI Ahmed Alameldin∗ School of Computing Clemson University aalamel@clemson.edu Ashton Williamson∗ School of Computing Clemson University taw2@clemson.edu Abstract This paper presents our system descriptions for SemEval 2023-Task 7: Multi-evidence Natu- ral Language Inference for Clinical Trial Data sub-tasks one and two. Provided with a collec- tion of Clinical Trial Reports (CTRs) and cor- responding expert-annotated claim statements, sub-task one involves determining an infer- ential relationship between the statement and CTR premise: contradiction or entailment. Sub- task two involves retrieving evidence from the CTR which is necessary to determine the en- tailment in sub-task one. For sub-task two we employ a recent transformer-based lan- guage model pretrained on biomedical litera- ture, which we domain-adapt on a set of clini- cal trial reports. For sub-task one, we take an ensemble approach in which we leverage the evidence retrieval model from sub-task two to extract relevant sections, which are then passed to a second model of equivalent architecture to determine entailment. Our system achieves a ranking of seventh on sub-task one with an F1- score of 0.705 and sixth on sub-task two with an F1-score of 0.806. In addition, we find that the high rate of success of language models on this dataset may be partially attributable to the existence of annotation artifacts. 1 1 Introduction The proliferation of Clinical Trial Reports (CTRs) has made it challenging for medical practitioners to stay up-to-date on the latest literature, inhibiting their ability provide evidence-based patient care. The computerized clinical decision support sys- tem (CDSS) has aimed to address this problem by providing an interface, typically within the elec- tronic health record system, with which a clinical practitioner can, for example, access automated pa- tient care recommendations or view diagnostic sug- gestions (Sutton et al., 2020). Research suggests 1Pretrained model weights are available at https:// huggingface.co/AshtonIsNotHere *These authors contributed equally to this work. that these systems can have a positive impact on both clinical practitioners and patients (Kwan et al., 2020; Sutton et al., 2020); they depend, however, on the ability to interpret and retrieve information from biomedical literature (Sutton et al., 2020). In this context, Task 7 provides a chance to augment existing approaches for retrieval and interpretation of unstructured biomedical text. Task 7 is split into two sub-tasks which are struc- tured around a dataset of English-language breast cancer CTRs: sub-task 1: textual entailment and sub-task 2: evidence retrieval. Sub-task one in- volves identifying the entailment relation (entailed or contradicted) between a statement–a claim about the CTR(s) annotated by a domain-expert–and a premise in the form of a section from the CTR(s). Sub-task 2 involves extracting sections of text from the CTR section which are necessary to either en- tail or contradict the statement. These tasks prove challenging by requiring modeling of long-term de- pendencies, comparisons between multiple CTRs, and complex numerical …

Content truncated to 500 words