Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction

Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction Hui-Syuan Yeh, Thomas Lavergne, Pierre Zweigenbaum Université Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Numérique, Orsay, France {yeh, lavergne, pz}@lisn.fr Abstract Relation extraction is a core problem for natural language processing in the biomedical domain. Recent research on relation extraction showed that prompt-based learning improves the performance on both fine-tuning on full training set and few-shot training. However, less effort has been made on domain-specific tasks where good prompt design can be even harder. In this paper, we investigate prompting for biomedical relation extraction, with experiments on the ChemProt dataset. We present a simple yet effective method to systematically generate comprehensive prompts that reformulate the relation extraction task as a cloze-test task under a simple prompt formulation. In particular, we experiment with different ranking scores for prompt selection. With BioMed-RoBERTa-base, our results show that prompting-based fine-tuning obtains gains by 14.21 F1 over its regular fine-tuning baseline. Besides, we find prompt-based learning requires fewer training examples to make reasonable predictions. The results demonstrate the potential of our methods in such a domain- specific relation extraction task. 1. Introduction With the rapid growth of biomedical textual resources in sci- entific articles, clinical notes, patient forums, social media, and so on, helping humans quickly grasp the key information out of vast content has become necessary. Natural Language Processing and more specifically Information Extraction (IE) algorithms support readers by transforming unstruc- tured text into structured information of interest. Relation extraction (RE), as one of the most important IE tasks, fo- cuses on recognizing the relation types between two entities mentioned in a given sentence (e.g., given Alfred Hitchcock directed Psycho, identify that the relation between (Alfred Hitchcock, Psycho) is DirectorOf). The current state of the art in information extraction is ob- tained by Transformer models such as BERT (Devlin et al., 2019). Great success has been obtained by adapting BERT architectures to biomedical tasks by additional train- ing (BioBERT (Lee et al., 2020), ClinicalBERT (Alsentzer et al., 2019)), or by pretraining from scratch (SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al., 2020)) on biomedical text corpora. More recently, BioMegatron (Shin et al., 2020a) studied the pretraining settings better for the biomedical BERT models; CharacterBERT (El Boukkouri et al., 2020) enabled word representations without requir- ing segmentation into a priori word pieces, to better repre- sent domain-specific terms in specialized domains. Another stream works on incorporating external knowledge bases into models (Michalopoulos et al., 2021; Liu et al., 2021a). Compared to previous work that augments training data with biomedical textual or structured data, we explore an alter- native training paradigm, prompting, to adapt pre-trained models to biomedical RE tasks more efficiently. The current dominant paradigm consists in pre-training a neural model with a language modeling objective such as masked word prediction (Devlin et al., 2019), then fine-tuning this model by retraining it with a different objective related to the target task (e.g., relation extraction). The main idea of prompt- ing, on …

Content truncated to 500 words