Revisiting the Negative Data of Distantly Supervised Relation Extraction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3572–3581 August 1–6, 2021. ©2021 Association for Computational Linguistics 3572 Revisiting the Negative Data of Distantly Supervised Relation Extraction Chenhao Xie1,2, Jiaqing Liang1,2, Jingping Liu1, Chengsong Huang1, Wenhao Huang1, Yanghua Xiao1,3∗ 1Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University 2Shuyan Technology Inc., Shanghai, China 3Fudan-Aishu Cognitive Intelligence Joint Research Center, Shanghai, China {redreamality, l.j.q.light}@gmail.com {jpliu17, huangcs19, whhuang17, shawyh}@fudan.edu.cn Abstract Distantly supervision automatically generates plenty of training samples for relation extrac- tion. However, it also incurs two major prob- lems: noisy labels and imbalanced training data. Previous works focus more on reduc- ing wrongly labeled relations (false positives) while few explore the missing relations that are caused by incompleteness of knowledge base (false negatives). Furthermore, the quantity of negative labels overwhelmingly surpasses the positive ones in previous problem formula- tions. In this paper, we first provide a thorough analysis of the above challenges caused by neg- ative data. Next, we formulate the problem of relation extraction into as a positive unla- beled learning task to alleviate false negative problem. Thirdly, we propose a pipeline ap- proach, dubbed RERE, that first performs sen- tence classification with relational labels and then extracts the subjects/objects. Experimen- tal results show that the proposed method con- sistently outperforms existing approaches and remains excellent performance even learned with a large quantity of false positive samples. Source code is available online1. 1 Introduction Relational extraction is a crucial step towards knowledge graph construction. It aims at identify- ing relational triples from a given sentence in the form of ⟨subject, relation, object⟩, in short, ⟨s, r, o⟩. For example, given S1 in Figure 1, we hope to ex- tract ⟨WILLIAM SHAKESPEARE, BIRTHPLACE, STRATFORD-UPON-AVON⟩. This task is usually modeled as a supervised learning problem and distant supervision (Mintz et al., 2009) is utilized to acquire large-scale train- ing data. The core idea is to obtain training data ∗Corresponding author 1https://github.com/redreamality/ RERE-relation-extraction Figure 1: Illustration of distant supervision process. S2- S5 are examples for four kinds of label noise. TP, FP, FN and PL mean true positive, false positive, false negative and partially labeled, respectively. “R-” or “E-” indi- cates whether the error occurs at relation-level or entity- level. Bold tokens are ground-truth subjects/objects. Underlined tokens together with the relation in the third column are labeled by distant supervision. “NA” means no relation. is through automatically labeling a sentence with existing relational triples from a knowledge base (KB). For example, given a triple ⟨s, r, o⟩ and a sentence, if the sentence contains both s and o, dis- tant supervision methods regard ⟨s, r, o⟩ as a valid sample for the sentence. If no relational triples are applicable, the sentence is labeled as “NA”. Despite the abundant training data obtained with distant supervision, nonnegligible errors also occur in the labels. There are two types of errors. In the first type, the labeled relation does …

Content truncated to 500 words