CaRB: A Crowdsourced Benchmark for Open IE

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6262–6267, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6262 CaRB: A Crowdsourced Benchmark for Open IE Sangnie Bhardwaj∗† Indian Institute of Technology New Delhi, India sangnie321@gmail.com Samarth Aggarwal∗ Indian Institute of Technology New Delhi, India samarthaggarwal2510@gmail.com Mausam Indian Institute of Technology New Delhi, India mausam@cse.iitd.ac.in Abstract Open Information Extraction (Open IE) sys- tems have been traditionally evaluated via manual annotation. Recently, an automated evaluator with a benchmark dataset (OIE2016) was released – it scores Open IE systems au- tomatically by matching system predictions with predictions in the benchmark dataset (Stanovsky and Dagan, 2016). Unfortunately, our analysis reveals that its data is rather noisy, and the tuple matching in the evaluator has is- sues, making the results of automated compar- isons less trustworthy. We contribute CaRB, an improved dataset and framework for testing Open IE systems. To the best of our knowledge, CaRB is the first crowdsourced Open IE dataset and it also makes substantive changes in the match- ing code and metrics. NLP experts anno- tate CaRB’s dataset to be more accurate than OIE2016. Moreover, we find that on one pair of Open IE systems, CaRB framework pro- vides contradictory results to OIE2016. Hu- man assessment verifies that CaRB’s ranking of the two systems is the accurate ranking. We release the CaRB framework along with its crowdsourced dataset. 1 Introduction Open Information Extraction (Open IE) refers to the task of forming relational tuples from sen- tences, without a fixed relation vocabulary (Banko et al., 2007). Open IE has numerous down- stream applications such as knowledge base con- struction, relation extraction, summarisation and learning word embeddings (Stanovsky et al., 2015; Mausam, 2016). There have been many Open IE systems till date such as TextRunner (Banko et al., 2007), ReVerb (Fader et al., 2011; Etzioni et al., 2011), OLLIE (Mausam et al., 2012), ClausIE * Joint first author † Presently an AI Resident at Google (Del Corro and Gemulla, 2013), OpenIE 4 (Chris- tensen et al., 2011; Pal and Mausam, 2016), Ope- nIE 5 (Saha et al., 2017; Saha and Mausam, 2018), PropS (Stanovsky et al., 2016), NST (Jia et al., 2018), Neural Open IE (Cui et al., 2018), and more. With the advent of so many systems, it is imperative to have a standardized mechanism for automatic evaluation so that they can be com- pared. Traditionally, these systems have been evalu- ated over small manually curated gold datasets (e.g., (Fader et al., 2011; Mausam et al., 2012)). There are two problems with this approach. One, it is not reliable due to the small size of annota- tion. Second, it lacks standardization, since there is no single gold dataset over which all systems are evaluated. Moreover, the guidelines to annotate may vary across datasets and annotators. Recently, some standard benchmarks datasets and evaluators have been proposed: OIE2016 (Stanovsky and Da- gan, 2016), RelVis (Schneider et …

Content truncated to 500 words