Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words

Proceedings of the 29th International Conference on Computational Linguistics, pages 1422–1431 October 12–17, 2022. 1422 Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words Haochun Wang1, Chi Liu1, Nuwa Xi1, Sendong Zhao1∗, Meizhi Ju2, Shiwei Zhang2 Ziheng Zhang2, Yefeng Zheng2, Bing Qin1 and Ting Liu1 1Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, China 2Tencent Jarvis Lab, Shenzhen, China {hcwang,cliu,nwxi,sdzhao,bqin,tliu}@ir.hit.edu.cn meizhi.ju@outlook.com, {zswvizhang,zihengzhang,yefengzheng}@tencent.com Abstract Prompt-based fine-tuning for pre-trained mod- els has proven effective for many natural lan- guage processing tasks under few-shot settings in general domain. However, tuning with prompt in biomedical domain has not been investigated thoroughly. Biomedical words are often rare in general domain, but quite ubiquitous in biomedical contexts, which dra- matically deteriorates the performance of pre- trained models on downstream biomedical ap- plications even after fine-tuning, especially in low-resource scenarios. We propose a sim- ple yet effective approach to helping models learn rare biomedical words during tuning with prompt. Experimental results show that our method can achieve up to 6% improvement in biomedical natural language inference task without any extra parameters or training steps using few-shot vanilla prompt settings. 1 Introduction Pre-trained models have achieved a great success in natural language processing (NLP) and become a new paradigm for various tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Qiu et al., 2020). Many studies have paid attention to pre- trained models in biomedical NLP tasks (Lee et al., 2020; Lewis et al., 2020; Zhao et al., 2021). How- ever, plain pre-trained models sometimes cannot do very well in biomedical NLP tasks. In general, there are two challenges to fully exploit the poten- tial of the pre-trained models for biomedical NLP tasks, i.e., (1) limited data and (2) rare biomed- ical words. Firstly, it is common that the amount of biomedical labeled data is limited due to strict privacy policy constraints (Šuster et al., 2017), high cost and professional requirement for data annota- tion. Pre-trained models perform poorly with few samples since abundant training samples are essen- tial to optimize task-related parameters (Liu et al., ∗Corresponding author 2021a). Secondly, biomedical words are usually low-frequency words but critical to understanding biomedical texts. As an example of natural lan- guage inference (NLI) task in Figure 1, the model goes wrong during tuning when faced with a rare word “afebrile”1 in the premise, whose meaning is “having no fever”. It can be no easy for the pre- trained models to predict the correct label if the models haven’t seen the rare biomedical words for enough times during pre-training or tuning stage. Thus, pre-trained models cannot capture the pre- cise semantics of biomedical texts in the scenario of low-resource tasks. With very few annotated samples available for a new task, it is hard to effectively fine-tune pre- trained models with the additional task-specific parameters, which is even more of a challenge to biomedical domain as mentioned above. Prompt technique has been introduced to smooth the fine- tuning process in the few-shot …

Content truncated to 500 words