Effectiveness and Efficiency of Open Relation Extraction

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 447–457, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Effectiveness and Efficiency of Open Relation Extraction Filipe Mesquita Jordan Schmidek Denilson Barbosa Department of Computing Science, University of Alberta, Canada {mesquita,schmidek,denilson}@ualberta.ca Abstract A large number of Open Relation Extrac- tion approaches have been proposed recently, covering a wide range of NLP machinery, from “shallow” (e.g., part-of-speech tagging) to “deep” (e.g., semantic role labeling–SRL). A natural question then is what is the trade- off between NLP depth (and associated com- putational cost) versus effectiveness. This pa- per presents a fair and objective experimental comparison of 8 state-of-the-art approaches over 5 different datasets, and sheds some light on the issue. The paper also describes a novel method, EXEMPLAR, which adapts ideas from SRL to less costly NLP machinery, resulting in substantial gains both in efficiency and ef- fectiveness, over binary and n-ary relation ex- traction tasks. 1 Introduction Open Relation Extraction (ORE) (Banko and Et- zioni, 2008) has become prevalent over traditional relation extraction methods, especially on the Web, because of the intrinsic difficulty in training indi- vidual extractors for every single relation. Broadly speaking, existing ORE approaches can be grouped according to the level of sophistication of the NLP techniques they rely upon: (1) shallow parsing, (2) dependency parsing and (3) semantic role labelling (SRL). Shallow methods annotate the sentences with part-of-speech (POS) tags and the ORE approaches in this category, such as ReVerb (Fader et al., 2011) and SONEX (Merhav et al., 2012), identify rela- tions by matching patterns over such tags. Depen- dency parsing gives unambiguous relations among each word in the sentence, and the ORE approaches in this category such as PATTY (Nakashole et al., 2012), OLLIE (Mausam et al., 2012), and TreeK- ernel (Xu et al., 2013) identify whole subtrees con- necting the relation predicate and its arguments. Fi- nally, semantic annotators, such as Lund (Johans- son and Nugues, 2008) and SwiRL (Surdeanu et al., 2003), add roles to each node in a parse tree, en- abling ORE approaches that identify the precise con- nection between each argument and the predicate in a relation, independently. The first contribution of the paper is an objec- tive and fair experimental comparison of the state- of-the-art in ORE, on 5 datasets with varying de- gree of “difficulty”. Of these, 4 datasets were an- notated manually, covering both well-formed sen- tences, from the New York Times (NYT) and the Penn Treebank, as well as mixed-quality sentences from a popular Web corpus. A much larger corpus with 12,000 sentences from NYT, automatically an- notated is also used. Another experiment focuses on n-ary relation extractions separately. The results show, as expected, that the three broad classes above are separated by orders of magnitude when it comes to throughput. Shallow methods handle ten times more sentences than dependency parsing methods, which in turn handle ten times more sentences than semantic parsing methods. Nevertheless, the cost- benefit trade-off is …

Content truncated to 500 words