Compensating for Annotation Errors in Training a Relation Extractor

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 194–203, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Compensating for Annotation Errors in Training a Relation Extractor Bonan Min Ralph Grishman New York University New York University 715 Broadway, 7th floor 715 Broadway, 7th floor New York, NY 10003 USA New York, NY 10003 USA min@cs.nyu.edu grishman@cs.nyu.edu Abstract The well-studied supervised Relation Extraction algorithms require training data that is accurate and has good coverage. To obtain such a gold standard, the common practice is to do independent double annotation followed by adjudication. This takes significantly more human effort than annotation done by a single annotator. We do a detailed analysis on a snapshot of the ACE 2005 annotation files to understand the differences between single-pass annotation and the more expensive nearly three-pass process, and then propose an algorithm that learns from the much cheaper single-pass annotation and achieves a performance on a par with the extractor trained on multi-pass annotated data. Furthermore, we show that given the same amount of human labor, the better way to do relation annotation is not to annotate with high-cost quality assurance, but to annotate more. 1. Introduction Relation Extraction aims at detecting and categorizing semantic relations between pairs of entities in text. It is an important NLP task that has many practical applications such as answering factoid questions, building knowledge bases and improving web search. Supervised methods for relation extraction have been studied extensively since rich annotated linguistic resources, e.g. the Automatic Content Extraction1 (ACE) training corpus, were released. We will give a summary of related methods in section 2. Those methods rely on accurate and complete annotation. To obtain high quality annotation, the common wisdom is to let 1 http://www.itl.nist.gov/iad/mig/tests/ace/ two annotators independently annotate a corpus, and then asking a senior annotator to adjudicate the disagreements 2. This annotation procedure roughly requires 3 passes3 over the same corpus. Therefore it is very expensive. The ACE 2005 annotation on relations is conducted in this way. In this paper, we analyzed a snapshot of ACE training data and found that each annotator missed a significant fraction of relation mentions and annotated some spurious ones. We found that it is possible to separate most missing examples from the vast majority of true-negative unlabeled examples, and in contrast, most of the relation mentions that are adjudicated as incorrect contain useful expressions for learning a relation extractor. Based on this observation, we propose an algorithm that purifies negative examples and applies transductive inference to utilize missing examples during the training process on the single-pass annotation. Results show that the extractor trained on single-pass annotation with the proposed algorithm has a performance that is close to an extractor trained on the 3-pass annotation. We further show that the proposed algorithm trained on a single-pass annotation on the complete set of documents has a higher performance than an extractor trained on 3-pass annotation on 90% of the documents in …

Content truncated to 500 words