Open Information Extraction on Scientific Text: An Evaluation

Proceedings of the 27th International Conference on Computational Linguistics, pages 3414–3423 Santa Fe, New Mexico, USA, August 20-26, 2018. 3414 Open Information Extraction on Scientific Text: An Evaluation Paul Groth Elsevier Labs 1600 John F. Kennedy Blvd. Suite 1800 Philadelphia, PA p.groth@elsevier.com Mike Lauruhn Elsevier Labs 1600 John F. Kennedy Blvd. Suite 1800 Philadelphia, PA m.lauruhn@elsevier.com Antony Scerri Elsevier Labs 1600 John F. Kennedy Blvd. Suite 1800 Philadelphia, PA a.scerri@elsevier.com Ron Daniel, Jr. Elsevier Labs 1600 John F. Kennedy Blvd. Suite 1800 Philadelphia, PA r.daniel@elsevier.com Abstract Open Information Extraction (OIE) is the task of the unsupervised creation of structured informa- tion from text. OIE is often used as a starting point for a number of downstream tasks including knowledge base construction, relation extraction, and question answering. While OIE methods are targeted at being domain independent, they have been evaluated primarily on newspaper, en- cyclopedic or general web text. In this article, we evaluate the performance of OIE on scientific texts originating from 10 different disciplines. To do so, we use two state-of-the-art OIE systems using a crowd-sourcing approach. We find that OIE systems perform significantly worse on sci- entific text than encyclopedic text. We also provide an error analysis and suggest areas of work to reduce errors. Our corpus of sentences and judgments are made available. 1 Introduction The scientific literature is growing at a rapid rate (Bornmann and Mutz, 2015). To make sense of this flood of literature, for example, to extract cancer pathways (Poon et al., 2014) or find geological fea- tures (Leveling, 2015), increasingly requires the application of natural language processing. Given the di- versity of information and its constant flux, the use of unsupervised or distantly supervised techniques are of interest (Quirk and Poon, 2017). In this paper, we investigate one such unsupervised method, namely, Open Information Extraction (OIE) (Banko et al., 2007). OIE is the task of the unsupervised creation of structured information from text. OIE is often used as a starting point for a number of downstream tasks including knowledge base construction, relation extraction, and question answering (Mausam, 2016). While OIE has been applied to the scientific literature before (Groth et al., 2016), we have not found a systematic evaluation of OIE as applied to scientific publications. The most recent evaluations of OIE extraction tools (Gashteovski et al., 2017; Schneider et al., 2017) have instead looked at the performance of these tools on traditional NLP information sources (i.e. encyclopedic and news-wire text). Indeed, as (Schneider et al., 2017) noted, there is little work on the evaluation of OIE systems. Thus, the goal of this paper is to evaluate the performance of the state of the art in OIE systems on scientific text. Specifically, we aim to test two hypotheses: 1. H1: There is no significant difference in the accuracy of OIE systems on scientific vs. general audience content. 2. H2: There is no significant difference in performance of current state-of-the-art OIE systems on scientific and medical content. Additionally, we seek to gain insight into …

Content truncated to 500 words