End-to-End Construction of NLP Knowledge Graph

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1885–1895 August 1–6, 2021. ©2021 Association for Computational Linguistics 1885 End-to-End NLP Knowledge Graph Construction Ishani Mondal∗ Microsoft Research Bengaluru, India ishani340@gmail.com Yufang Hou IBM Research Europe Dublin, Ireland yhou@ie.ibm.com Charles Jochim IBM Research Europe Dublin, Ireland Abstract This paper studies the end-to-end construction of an NLP Knowledge Graph (KG) from scien- tific papers. We focus on extracting four types of relations: evaluatedOn between tasks and datasets, evaluatedBy between tasks and eval- uation metrics, as well as coreferent and re- lated relations between the same type of en- tities. For instance, “F1 score” is coreferent with “F-measure”. We introduce novel meth- ods for each of these relation types and apply our final framework (SciNLP-KG) to 30,000 NLP papers from ACL Anthology to build a large-scale KG, which can facilitate automat- ically constructing scientific leaderboards for the NLP community. The results of our exper- iments indicate that the resulting KG contains high-quality information. 1 Introduction As interest in the NLP research community grows, the number of NLP tasks, datasets, and metrics for evaluation also grows, making it increasingly dif- ficult for researchers to keep track of the plethora of new resources. In order to tackle this problem, recently there have been a few manual efforts to summarize the state-of-the-art on selected subfields of NLP in the form of leaderboards that extract tasks, datasets, metrics and results from papers, such as NLP-progress1 or Paperswithcode2. How- ever, these manual efforts are not sustainable over time for all NLP tasks. Meanwhile, although there are various studies focusing on extracting entities and relations from scientific literature (Augenstein and Søgaard, 2017; Luan et al., 2018; G´abor et al., 2018; Hou et al., 2019; Jain et al., 2020), there is less work on con- structing KGs that contain rich information about ∗Work done during internship at IBM Research. 1https://github.com/sebastianruder/NLP-progress 2https://paperswithcode.com tasks, datasets, and metrics. Such a KG would be highly beneficial for the researchers to understand the underlying related literature for a particular task, or to perform comparable experiments. In this paper, we propose an end-to-end approach to construct a KG from the NLP papers. Our KG contains three types of entities (tasks, datasets, and metrics) and four types of relations connecting them. Figure 1 (bottom left) depicts these rela- tions in our large-scale graph built from 30,000 NLP papers from the ACL Anthology. For instance, “sentiment analysis (task)” is evaluatedOn “IMDb dataset”, “opinion analysis (task)” is evaluatedBy “Precision (metric)”, “F1-score (metric)” is corefer- ent with “F1-measure (metric)”, and “YELP review dataset” is related to “IMDb dataset”. We develop a framework (SciNLP-KG, Section 5) to extract these relations based on NLP papers that are tagged for Task (T), Dataset (D), and Met- ric (M) entities using a TDM entity tagger. Our framework primarily consists of three learning- based models. First, motivated by Hou et al.’s work on tagging NLP papers with valid TDM triples based on a small manually created gold TDM tax- onomy, we design a …

Content truncated to 500 words