Effective Crowd Annotation for Relation Extraction

Proceedings of NAACL-HLT 2016, pages 897–906, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Effective Crowd Annotation for Relation Extraction Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, and Daniel S. Weld Turing Center, Department of Computer Science and Engineering Box 352350 University of Washington Seattle, WA 98195, USA {anglil, soderlan, jbragg, chrislin, xiaoling, weld} at cs.washington.edu Abstract Can crowdsourced annotation of training data boost performance for relation extraction over methods based solely on distant supervision? While crowdsourcing has been shown effec- tive for many NLP tasks, previous researchers found only minimal improvement when ap- plying the method to relation extraction. This paper demonstrates that a much larger boost is possible, e.g., raising F1 from 0.40 to 0.60. Furthermore, the gains are due to a sim- ple, generalizable technique, Gated Instruc- tion, which combines an interactive tutorial, feedback to correct errors during training, and improved screening. 1 Introduction Relation extraction (RE) is the task of identifying instances of relations, such as nationality (person, country) or place of birth (person, location), in pas- sages of natural text. Since RE enables a broad range of applications — including question answering and knowledge base population — it has attracted atten- tion from many researchers. Many approaches to RE use supervised machine learning, e.g., (Soderland et al., 1995; Califf and Mooney, 1997; Lafferty et al., 2001), but these methods require a large, human- annotated training corpus that may be unavailable. In response, researchers developed methods for distant supervision (DS) in which a knowledge base such as Wikipedia or Freebase is used to automati- cally tag training examples from a text corpus (Wu and Weld, 2007; Mintz et al., 2009). Indeed, virtu- ally all entries to recent TAC KBP relation extraction 0.0 0.2 0.4 0.6 0.8 1.0 Distantly Supervised (DS) Data 0.0 0.2 0.4 0.6 0.8 1.0 DS Data Plus 20k GI Crowdsourced Examples precision nationality precision born precision lived precision died precision traveled recall nationality recall born recall lived recall died recall traveled F1 nationality F1 born F1 lived F1 died F1 traveled Figure 1: Adding 20K crowdsourced instances, acquired using Gated Instruction, to 700K examples from distant supervision raises precision, recall, and F1 for nearly all relations and raises overall F1 from 0.40 to 0.60 with MIML-RE learning. competitions use distant supervision (Ji and Grish- man, 2011). However, distant supervision provides noisy training data with many false positives, and this limits the precision of the resulting extractors (see Section 2). A natural assumption is that human- annotated training data, either alone or in conjunc- tion with distant supervision, would give better pre- cision. In particular, Snow et al. (2008) showed that, for many NLP tasks, crowdsourced data is as good as or better than that annotated by experts. It is quite surprising, therefore, that researchers who have applied crowdsourced annotation to rela- tion extraction argue the opposite, that crowdsourc- ing provides only minor improvement: 897 • Zhang et al. (2012) conclude that “Human feed- back has relatively …

Content truncated to 500 words