Extreme Extraction – Machine Reading in a Week

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1437–1446, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Extreme Extraction -- Machine Reading in a Week Marjorie Freedman, Lance Ramshaw, Elizabeth Boschee, Ryan Gabbard, Gary Kratkiewicz, Nicolas Ward, Ralph Weischedel Raytheon BBN Technologies 10 Moulton St. Cambridge, MA 02138 mfreedma,lramshaw,eboschee,rgabbard,kratkiewicz, nward,weischedel@bbn.com The views expressed are those of the author and do not reflect the official policy or position of the Depart- ment of Defense or the U.S. Government. This is in accordance with DoDI 5230.29, January 8, 2009. Abstract We report on empirical results in extreme extraction. It is extreme in that (1) from re- ceipt of the ontology specifying the target concepts and relations, development is li- mited to one week and that (2) relatively little training data is assumed. We are able to surpass human recall and achieve an F1 of 0.51 on a question-answering task with less than 50 hours of effort using a hybrid approach that mixes active learning, boot- strapping, and limited (5 hours) manual rule writing. We compare the performance of three systems: extraction with handwrit- ten rules, bootstrapped extraction, and a combination. We show that while the recall of the handwritten rules surpasses that of the learned system, the learned system is able to improve the overall recall and F1. 1 Introduction Throughout the Automatic Content Extraction 1 (ACE) evaluations and the Message Understanding Conferences2 (MUC), teams typically had a year or more from release of the target to submitting sys- tem results. One exception was MUC-6 (Grishman & Sundheim, 1996), in which scenario templates for changing positions were extracted given only one month. Our goal was to confine development to a calendar week, in fact, <50 person hours. This 1 http://www.nist.gov/speech/tests/ace/ 2 http://www-nlpir.nist.gov/related_projects/muc/ is significant in two ways: the less effort it takes to bring up a new domain, (1) the more broadly ap- plicable the technology is and (2) the less effort required to run a diagnostic research experiment. Our second goal concerned minimizing training data. Rather than approximately 250k words of entity and relation annotation as in ACE, only ~20 example pairs per relation-type were provided as training. Reducing the training requirements has the same two desirable outcomes: demonstrating that the technology can be broadly applicable and reducing the overhead for running experiments. The system achieved recall of 0.49 and precision of 0.53 (for an F1 of 0.51) on a blind test set of 60 queries of the form Ri(arg1, arg2), where Ri is one of the 5 new relations and exactly one of arg1 or arg2 is a free variable for each query.  Key to this achievement was a hybrid of: a variant of (Miller, et al., 2004) to learn two new classes of entities via automatically induced word classes and active learning (6 hours)  bootstrap relation learning (Freedman et al, 2010) to learn 5 new relation classes (2.5 hours),  handwritten patterns over predicate-argument structure (5 hours), and  …

Content truncated to 500 words