Combining Distant and Partial Supervision for Relation Extraction

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1556–1567, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Combining Distant and Partial Supervision for Relation Extraction Gabor Angeli, Julie Tibshirani, Jean Y. Wu, Christopher D. Manning Stanford University Stanford, CA 94305 {angeli, jtibs, jeaneis, manning}@stanford.edu Abstract Broad-coverage relation extraction either requires expensive supervised training data, or suffers from drawbacks inherent to distant supervision. We present an ap- proach for providing partial supervision to a distantly supervised relation extrac- tor using a small number of carefully se- lected examples. We compare against es- tablished active learning criteria and pro- pose a novel criterion to sample examples which are both uncertain and representa- tive. In this way, we combine the ben- efits of fine-grained supervision for diffi- cult examples with the coverage of a large distantly supervised corpus. Our approach gives a substantial increase of 3.9% end- to-end F1 on the 2013 KBP Slot Filling evaluation, yielding a net F1 of 37.7%. 1 Introduction Fully supervised relation extractors are limited to relatively small training sets. While able to make use of much more data, distantly supervised ap- proaches either make dubious assumptions in or- der to simulate fully supervised data, or make use of latent-variable methods which get stuck in local optima easily. We hope to combine the benefits of supervised and distantly supervised methods by annotating a small subset of the available data us- ing selection criteria inspired by active learning. To illustrate, our training corpus contains 1 208 524 relation mentions; annotating all of these mentions for a fully supervised classifier, at an average of $0.13 per annotation, would cost ap- proximately $160 000. Distant supervision allows us to make use of this large corpus without requir- ing costly annotation. The traditional approach is based on the assumption that every mention of an entity pair (e.g., Obama and USA) participates in the known relation between the two (i.e., born in). However, this introduces noise, as not every men- tion expresses the relation we are assigning to it. We show that by providing annotations for only 10 000 informative examples, combined with a large corpus of distantly labeled data, we can yield notable improvements in performance over the distantly supervised data alone. We report results on three criteria for selecting examples to anno- tate: a baseline of sampling examples uniformly at random, an established active learning criterion, and a new metric incorporating both the uncer- tainty and the representativeness of an example. We show that the choice of metric is important – yielding as much as a 3% F1 difference – and that our new proposed criterion outperforms the standard method in many cases. Lastly, we train a supervised classifier on these collected exam- ples, and report performance comparable to dis- tantly supervised methods. Furthermore, we no- tice that initializing the distantly supervised model using this supervised classifier is critical for ob- taining performance improvements. This work makes a number of concrete contri- butions. …

Content truncated to 500 words