Saama AI Research at SemEval-2023 Task 7: Exploring the Capabilities of Flan-T5 for Multi-evidence Natural Language Inference in Clinical Trial Data

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 995–1003 July 13-14, 2023 ©2023 Association for Computational Linguistics Saama AI Research at SemEval-2023 Task 7: Exploring the Capabilities of Flan-T5 for Multi-evidence Natural Language Inference in Clinical Trial Data Kamal Raj Kanakarajan and Malaikannan Sankarasubbu SAAMA AI Research Lab, Chennai, India {kamal.raj, malaikannan.sankarasubbu}@saama.com Abstract The goal of the NLI4CT task is to build a Nat- ural Language Inference system for Clinical Trial Reports that will be used for evidence interpretation and retrieval. Large Language models have demonstrated state-of-the-art per- formance in various natural language process- ing tasks across multiple domains. We suggest using an instruction-finetuned Large Language Models (LLMs) to take on this particular task in light of these developments. We have evaluated the publicly available LLMs under zeroshot set- ting, and finetuned the best performing Flan-T5 model for this task. On the leaderboard, our sys- tem ranked second, with an F1 Score of 0.834 on the official test set. 1 Introduction Building a Natural Language Inference (NLI) sys- tem for Clinical Trial Reports (CTRs) is the objec- tive of the Multi-evidence Natural Language Infer- ence for Clinical Trial Data (NLI4CT) task Jullien et al. (2023), which focuses on the interpretation and retrieval of medical evidence. Clinical practi- tioners struggle to keep up with the large number of CTRs published yearly to provide individualised, evidence-based care DeYoung et al. (2020). The main task is separated into two subtasks. Task 1 is to determine the relationship of the claim to the in- formation in a single CTR or to compare two CTRs to see if there is entailment or contradiction. The supporting factor required to back up the predicted label in Task 1 must be extracted from the CTRs in Task 2. English is the only language used in the task. BERT Devlin et al. (2019), ULMFiT Howard and Ruder (2018), and GPT Radford et al. (2018) models revolutionised the field of Natural Lan- guage Processing by introducing a new technique of pretraining on Language modelling and fine- tuning on task-specific data using supervised data, resulting in state-of-the-art results on a variety of datasets and benchmarks Wang et al. (2018), Wang et al. (2019). When applying the same model ar- chitecture to domains such as biomedical, they also achieved state-of-the-art results on domain specific tasks; domain specific trained models like BioBERT Lee et al. (2020), PubMedBERT Gu et al. (2021), BioELECTRA raj Kanakarajan et al. (2021) further improved performance. Furthermore, the introduction of Large Language Models (LLMs) GPT-3 Brown et al. (2020), PaLM Chowdhery et al. (2022), OPT Zhang et al. (2022), GLM Zeng et al. (2022), BLOOM Scao et al. (2022) ushered in a new strategy known as Prompt-based learning Liu et al. (2023), which allows these models to be more effectively adapted to tasks. Prompt-based learning is a technique for using LLM models to predict what will happen next in a sentence. The mod- els fill in the blanks to form the final sentence, which can …

Content truncated to 500 words