BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4472 - 4490 May 22-27, 2022 c⃝2022 Association for Computational Linguistics BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation Kiril Gashteovski1, Mingying Yu1,2, Bhushan Kotnis1, Carolin Lawrence1, Mathias Niepert1,4, Goran Glava˘s2,3 1NEC Laboratories Europe GmbH, Heidelberg, Germany 2University of Mannheim, 3LMU Munich, Germany 4University of Stuttgart, Germany firstname.lastname@neclab.eu goran@informatik.uni-mannheim.de Abstract Intrinsic evaluations of OIE systems are car- ried out either manually—with human evalu- ators judging the correctness of extractions— or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incom- pleteness of the existing OIE benchmarks: the ground truth extractions do not include all ac- ceptable variants of the same fact, leading to unreliable assessment of the models’ perfor- mance. Moreover, the existing OIE bench- marks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chi- nese, and German. In contrast to existing OIE benchmarks, BenchIE is fact-based, i.e., it takes into account informational equivalence of extractions: our gold standard consists of fact synsets, clusters in which we exhaustively list all acceptable surface forms of the same fact. Moreover, having in mind common downstream applications for OIE, we make BenchIE multi-faceted; i.e., we create bench- mark variants that focus on different facets of OIE evaluation, e.g., compactness or mini- mality of extractions. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are signifi- cantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available.1 1 Introduction Open Information Extraction (OIE) is the task of extracting relations and their arguments from natu- ral language text in a schema-free manner (Banko et al., 2007). Consider the sentence "Sen. Mitchell, who is from Maine, is a lawyer."; an OIE system is expected to extract the triples ("Sen. Mitchell"; "is from"; "Maine") and ("Sen. Mitchell"; "is"; "a lawyer") from the sentence. OIE systems are used 1https://github.com/gkiril/benchie in many downstream tasks, including knowledge graph (KG) population (Gashteovski et al., 2020), open link prediction (Broscheit et al., 2020), and question answering (Yan et al., 2018). These down- stream tasks lend themselves as natural setups for extrinsic OIE evaluation (Mausam, 2016). While valuable in concrete applications, such extrinsic evaluations do not measure the intrinsic correct- ness of the extracted facts: for that purpose, several benchmarks for intrinsic OIE evaluation have been proposed (Stanovsky and Dagan, 2016; Lechelle et al., 2019; Bhardwaj et al., 2019). Automated benchmark evaluations are more fea- sible (i.e., faster and cheaper) than manual OIE evaluations (Hohenecker et al., 2020). The cur- rent benchmarks, however, use scoring functions that are based on approximate (token-level) match- ing of system extractions against ground truth facts, which seems to be substantially less reli- able than human judgments of extraction correct- ness (Zhan and Zhao, 2020). This primarily stems from the incompleteness of existing OIE …

Content truncated to 500 words