Distant Supervision for Relation Extraction with an Incomplete Knowledge Base

Proceedings of NAACL-HLT 2013, pages 777–782, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Distant Supervision for Relation Extraction with an Incomplete Knowledge Base Bonan Min, Ralph Grishman, Li Wan New York University New York, NY 10003 {min,grishman,wanli} @cs.nyu.edu Chang Wang, David Gondek IBM T. J. Watson Research Center Yorktown Heights, NY 10598 {wangchan,dgondek} @us.ibm.com Abstract Distant supervision, heuristically labeling a corpus using a knowledge base, has emerged as a popular choice for training relation ex- tractors. In this paper, we show that a sig- nificant number of “negative“ examples gen- erated by the labeling process are false neg- atives because the knowledge base is incom- plete. Therefore the heuristic for generating negative examples has a serious flaw. Building on a state-of-the-art distantly-supervised ex- traction algorithm, we proposed an algorithm that learns from only positive and unlabeled labels at the pair-of-entity level. Experimental results demonstrate its advantage over existing algorithms. 1 Introduction Relation Extraction is a well-studied problem (Miller et al., 2000; Zhou et al., 2005; Kambhatla, 2004; Min et al., 2012a). Recently, Distant Super- vision (DS) (Craven and Kumlien, 1999; Mintz et al., 2009) has emerged to be a popular choice for training relation extractors without using manually labeled data. It automatically generates training ex- amples by labeling relation mentions1 in the source corpus according to whether the argument pair is listed in the target relational tables in a knowledge base (KB). This method significantly reduces human efforts for relation extraction. The labeling heuristic has a serious flaw. Knowl- edge bases are usually highly incomplete. For exam- 1An occurrence of a pair of entities with the source sentence. ple, 93.8% of persons from Freebase2 have no place of birth, and 78.5% have no nationality (section 3). Previous work typically assumes that if the argument entity pair is not listed in the KB as having a re- lation, all the corresponding relation mentions are considered negative examples.3 This crude assump- tion labeled many entity pairs as negative when in fact some of their mentions express a relation. The number of such false negative matches even exceeds the number of positive pairs, by 3 to 10 times, lead- ing to a significant problem for training. Previous approaches (Riedel et al., 2010; Hoffmann et al., 2011; Surdeanu et al., 2012) bypassed this problem by heavily under-sampling the “negative“ class. We instead deal with a learning scenario where we only have entity-pair level labels that are either posi- tive or unlabeled. We proposed an extension to Sur- deanu et al. (2012) that can train on this dataset. Our contribution also includes an analysis on the incom- pleteness of Freebase and the false negative match rate in two datasets of labeled examples generated by DS. Experimental results on a realistic and chal- lenging dataset demonstrate the advantage of the al- gorithm over existing solutions. 2 Related Work Distant supervision was first proposed by Craven and Kumlien (1999) in the biomedical domain. 2Freebase is a large collaboratively-edited KB. It is available at http://www.freebase.com. …

Content truncated to 500 words