Creating a Large Benchmark for Open Information Extraction

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2300–2305, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Creating a Large Benchmark for Open Information Extraction Gabriel Stanovsky and Ido Dagan Computer Science Department, Bar-Ilan University, Ramat Gan, Israel gabriel.satanovsky@gmail.com dagan@cs.biu.ac.il Abstract Open information extraction (Open IE) was presented as an unrestricted variant of tra- ditional information extraction. It has been gaining substantial attention, manifested by a large number of automatic Open IE extractors and downstream applications. In spite of this broad attention, the Open IE task definition has been lacking – there are no formal guide- lines and no large scale gold standard annota- tion. Subsequently, the various implementa- tions of Open IE resorted to small scale post- hoc evaluations, inhibiting an objective and re- producible cross-system comparison. In this work, we develop a methodology that lever- ages the recent QA-SRL annotation to create a first independent and large scale Open IE an- notation,1 and use it to automatically compare the most prominent Open IE systems. 1 Introduction Open Information Extraction (Open IE) was origi- nally formulated as a function from a document to a set of tuples indicating a semantic relation between a predicate phrase and its arguments (Banko et al., 2007). Wu and Weld (2008) further defined that an Open IE extractor should “produce one triple for ev- ery relation stated explicitly in the text, but is not required to infer implicit facts”. For example, given the sentence “John managed to open the door” an Open IE extractor should produce the tuple (John; managed to open; the door) but is not required to pro- duce the extraction (John; opened; the door). 1Publicly available at http://www.cs.biu.ac.il/ nlp/resources/downloads Following this initial presentation of the task, Open IE has gained substantial and consistent atten- tion. Many automatic extractors were created (e.g., (Fader et al., 2011; Mausam et al., 2012; Del Corro and Gemulla, 2013)) and were put to use in various downstream applications. In spite of this wide attention, Open IE’s for- mal definition is lacking. There are no clear guide- lines as to what constitutes a valid proposition to be extracted, and subsequently there is no large scale benchmark annotation. Open IE evaluations there- fore usually consist of a post-hoc manual evaluation of a small output sample. This evaluation practice lacks in several respects: (1) Most works provide a precision oriented metric, whereas recall is often not measured, (2) the num- bers are not comparable across systems, as they use different guidelines and datasets, and (3) the experi- ments are hard to replicate. In this work, we aim to contribute to the standard- ization of Open IE evaluation by providing a large gold benchmark corpus. For that end, we first iden- tify consensual guiding principles across prominent Open IE systems, resulting in a clearer formulation of the Open IE task. Following, we find that the re- cent formulation of QA-SRL (He et al., 2015) in fact subsumes these requirements for Open IE. This …

Content truncated to 500 words