Relation Extraction with Weighted Contrastive Pre-training on Distant Supervision

Findings of the Association for Computational Linguistics: EACL 2023, pages 2580–2585 May 2-6, 2023 ©2023 Association for Computational Linguistics Relation Extraction with Weighted Contrastive Pre-training on Distant Supervision Zhen Wan Fei Cheng Qianying Liu Zhuoyuan Mao Haiyue Song Kyoto University, JapanSadao Kurohashi {zhenwan, feicheng, ying, zhuoyuanmao, song, kuro} @nlp.ist.i.kyoto-u.ac.jp Abstract Contrastive pre-training on distant supervision has shown remarkable effectiveness in improv- ing supervised relation extraction tasks. How- ever, the existing methods ignore the intrin- sic noise of distant supervision during the pre-training stage. In this paper, we pro- pose a weighted contrastive learning method by leveraging the supervised data to estimate the reliability of pre-training instances and explicitly reduce the effect of noise. Exper- imental results on three supervised datasets demonstrate the advantages of our proposed weighted contrastive learning approach com- pared to two state-of-the-art non-weighted base- lines.Our code and models are available at: https://github.com/YukinoWan/WCL. 1 Introduction Relation extraction (RE) is the task of identifying the relationship between entities mentioned in the text, which can benefit many downstream tasks such as question answering and knowledge base population. Since most of the existing RE mod- els (Zhang et al., 2020; Zeng et al., 2020; Lin et al., 2020; Wang and Lu, 2020; Zhong and Chen, 2021) are trained on the labeled data, the amount of training data limits the performance of super- vised RE systems. To tackle this problem, recent work leverage semi-supervised distant supervision (DS) (Mintz et al., 2009; Lin et al., 2016; Vashishth et al., 2018; Chen et al., 2021) approach to gener- ate abundant training data by aligning knowledge bases (KBs) and raw corpora. However, distantly supervised relation extraction (DSRE) inevitably suffers from wrong labeling noise. Introducing a robust framework that utilizes both the abundant but noisy data from DS and the scarce but accu- rate data from human annotations becomes a new research line to improve RE systems. Recent works (Baldini Soares et al., 2019; Or- mándi et al., 2021; Peng et al., 2020) propose a Joe Biden is the 46th president of America. Joe Biden was born and raised in Scranton, Pennsylvania, America. Joe Biden graduated from the University of Delaware in America. … Triplet (Joe Biden, president of, America) in KB DS Label president of president of president of Unreliable positive samples Anchor Raw corpus Extracted instances Figure 1: An example of unreliable positive samples caused by DS noise. two-stage RE framework that they first design a RE-oriented task to pre-train BERT on DS data and then fine-tune on human-annotated (HA) datasets. Peng et al. (2020) use Wikipedia articles as the corpus and Wikidata as the KB in the pre-training stage to construct the DS data, and they introduce a contrastive learning-based method to pre-train BERT on the generated DS data. Given an anchor instance with a specific relation in the DS data, their contrastive learning method randomly selects one positive sample holding the same relation and max- imizes the similarity between the anchor and pos- itive sample. Meanwhile, their method randomly selects multiple negative …

Content truncated to 500 words