HW-TSC at SemEval-2023 Task 7: Exploring the Natural Language Inference Capabilities of ChatGPT and Pre-trained Language Model for Clinical Trial

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1603–1608 July 13-14, 2023 ©2023 Association for Computational Linguistics HW-TSC at SemEval-2023 Task 7: Exploring the Natural Language Inference Capabilities of ChatGPT and Pre-trained Language Model for Clinical Trial Xiaofeng Zhao, Min Zhang, MiaoMiao Ma, Chang Su, Yilun Liu, Minghan Wang, Xiaosong Qiao, Jiaxin Guo, Yinglu Li, Wenbing Ma, Shimin Tao, Hao Yang Huawei Translation Services Center, Beijing, China {zhaoxiaofeng14, zhangmin186, mamiaomiao, suchang8, liuyilun3 wangminghan, qiaoxiaosong, guojiaxin1, liyinglu, mawenbing taoshimin, yanghao30}@huawei.com Abstract In this paper, we describe a effective system for SemEval-2022 Task 7. This task aims to deter- mine whether a given statement is supported by comparing one or two clinical trial reports, and to identify evidence that supports the state- ment. This is a task that requires high natural language inference capabilities. In Subtask 1, we compare our strategy based on prompt learn- ing and ChatGPT with a baseline constructed using BERT in zero-shot setting, and validate the effectiveness of our strategy. In Subtask 2, we fine-tune DeBERTaV3 for classification without relying on the results from Subtask 1. We find that early stopping of the training can effectively prevent model overfitting, and this achieves a good performance in Subtask 2. In addition, we do not use any ensemble strategies. We have achieved the 10th place in Subtask 1 and the 2nd place in Subtask 2. 1 Introduction In recent years, the introduction of the pre-trained masked language model (LM) BERT (Devlin et al., 2018; Vaswani et al., 2017) has been a significant milestone in the field of natural language process- ing (NLP). BERT achieves an absolute improve- ment of 7.7 points on the General Language Un- derstanding Evaluation (GLUE) benchmark and produces state-of-the-art (SOTA) results in multi- ple tasks. Subsequently, numerous pre-trained lan- guage models (PLMs) based on the transformer architecture appear, such as XLNet (You et al., 2019), which addresses the issue of mask inde- pendence, BioBERT (Lee et al., 2020), a domain- specific model focused on biomedical text , and RoBERTa (Liu et al., 2019). More recently, in 2021, DeBERTa (He et al., 2020) has emerged as a powerful model that achieves scores surpassing hu- man performance on the SuperGLUE leaderboard, thanks to its disentangled attention and enhanced decoding mechanisms. In addition to models that reinforce natural language understanding, there are also models that focus on natural language gener- ation, such as GPT (Radford et al., 2018), GPT-2 (Radford et al., 2019), and GPT-3 (Brown et al., 2020). For example, in 2022, ChatGPT and In- structGPT (Ouyang et al., 2022) have gained popu- larity for their remarkable performance using rein- forcement learning and Proximal Policy Optimiza- tion. Their chain-of-thought (CoT) ability with super large models enables logical reasoning in zero-resource scenarios. 2 Task Description The goal of this task (Jullien et al., 2023) is to evaluate the natural language inference capability of systems on clinical trial reports in the medical domain. In recent years, there has been a significant increase in the publication of …

Content truncated to 500 words