BpHigh at SemEval-2023 Task 7: Can Fine-tuned Cross-encoders Outperform GPT-3.5 in NLI Tasks on Clinical Trial Data?

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1936–1944 July 13-14, 2023 ©2023 Association for Computational Linguistics BpHigh at SemEval-2023 Task 7: Can fine-tuned cross-encoders outperform GPT-3.5 in NLI tasks on clinical trial data? Bhavish Pahwa Mindtickle, Pune, India bhavishpahwa@gmail.com Bhavika Pahwa Government medical College, Aurangabad bhavikapahwa@gmail.com Abstract Many nations and organizations have begun collecting and storing clinical trial records for storage and analytical purposes so that medi- cal and clinical practitioners can refer to them on a centralized database over the internet and stay updated with the current clinical informa- tion. Clinical trial records have increased sig- nificantly, making it difficult for many medi- cal and clinical practitioners to stay updated with the latest information. To help and sup- port medical and clinical practitioners, there is a need to build intelligent systems that can update them with the latest information in a byte-sized condensed format and, at the same time, leverage their understanding capabilities to help them make decisions. This paper de- scribes our contribution to SemEval 2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT). Our results show that there is still a need to build domain- specific models as smaller transformer-based models can be finetuned on that data and outper- form foundational large language models like GPT-3.5. We also demonstrate how the per- formance of GPT-3.5 can be increased using few-shot prompting by leveraging the semantic similarity of the text samples and the few-shot train snippets. We will also release our code and models on open-source hosting platforms, GitHub and HuggingFace. 1 Introduction The Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) shared task (Jul- lien et al., 2023) aims at building clinical support systems on top of clinical trial reports to help clini- cal professionals. Clinical Trials are studies where the researchers test new ways to prevent, diagnose or treat a disease/disorder or find newer ways to im- prove the quality of life for people with chronic dis- eases. These studies are available as Clinical Trial Reports ( CTRs ), providing detailed study meth- ods and results. Clinical decision support systems can aid clinical and medical professionals in their day-to-day tasks and provide meaningful insights to complement their productivity. Clinical decision support systems consist of common frameworks and analytical software built on EHRs(Electronic health reports). However, the growth and devel- opment in AI and NLP opens new doors and op- portunities which entrepreneurs, researchers, and engineers can leverage to build next-gen intelligent systems which can leverage the contextual informa- tion of EHRs and CTRs. Nevertheless, one must be vigilant and take extra precautions while building intelligent systems for clinical domains, as it is a complex domain (Sutton et al., 2020). Recently many researchers have started building domain-specific versions of famous transformer architectures like BERT, GPT, and T5. Alsentzer et al. (2019) demonstrate how domain-specific Clin- ical BERT, Clinical BioBERT outperform standard BERT model on MedNLI natural language infer- ence task (Romanov and Shivade, 2018). However, there are certain …

Content truncated to 500 words