Continual Contrastive Finetuning Improves Low-Resource Relation Extraction

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13249–13263 July 9-14, 2023 ©2023 Association for Computational Linguistics Continual Contrastive Finetuning Improves Low-Resource Relation Extraction Wenxuan Zhou†, Sheng Zhang‡, Tristan Naumann‡, Muhao Chen† and Hoifung Poon ‡ †University of Southern California, ‡Microsoft Research {zhouwenx,muhaoche}@usc.edu {zhang.sheng,tristan,hoifung}@microsoft.com Abstract Relation extraction (RE), which has relied on structurally annotated corpora for model train- ing, has been particularly challenging in low- resource scenarios and domains. Recent lit- erature has tackled low-resource RE by self- supervised learning, where the solution in- volves pretraining the entity pair embedding by RE-based objective and finetuning on la- beled data by classification-based objective. However, a critical challenge to this approach is the gap in objectives, which prevents the RE model from fully utilizing the knowledge in pretrained representations. In this paper, we aim at bridging the gap and propose to pre- train and finetune the RE model using consis- tent objectives of contrastive learning. Since in this kind of representation learning paradigm, one relation may easily form multiple clusters in the representation space, we further pro- pose a multi-center contrastive loss that allows one relation to form multiple clusters to bet- ter align with pretraining. Experiments on two document-level RE datasets, BioRED and Re- DocRED, demonstrate the effectiveness of our method. Particularly, when using 1% end-task training data, our method outperforms PLM- based RE classifier by 10.5% and 6.1% on the two datasets, respectively. 1 Introduction Relation extraction (RE) is a fundamental task in NLP. It aims to identify the relations among en- tities in a given text from a predefined set of re- lations. While much effort has been devoted to RE in supervised settings (Zhang et al., 2017, 2018; Nan et al., 2020), RE is extremely challeng- ing in high-stakes domains such as biology and medicine, where annotated data are comparatively scarce due to overly high annotation costs. There- fore, there is a practical and urgent need for de- veloping low-resource RE models without the re- liance on large-scale end-task annotations. To realize low-resource RE, previous work has focused on pretraining entity pair embed- ding on large corpora using RE-based pretrain- ing objectives. Particularly, Baldini Soares et al. (2019) propose a self-supervised matching-the- blanks (MTB) objective that encourages embed- dings of the same entity pairs in different sen- tences to be similar. Later work (Peng et al., 2020; Qin et al., 2021) extends this idea with distant su- pervision (Mintz et al., 2009) and improves repre- sentation learning using contrastive learning (Had- sell et al., 2006; Oord et al., 2018; Chen et al., 2020). To adapt to training on RE annotations, these works finetune pretrained entity pair embed- ding on labeled data using classification-based ob- jectives. Although this paradigm produces better results compared to RE models initialized with pretrained language models (PLMs), it creates a significant divergence between pretraining and finetuning objectives, thus preventing the model from fully exploiting knowledge in pretraining. In this paper, we aim to bridge this …

Content truncated to 500 words