Reproducing Neural Ensemble Classifier for Semantic Relation Extraction inScientific Papers

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5569–5578 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5569 Reproducing Neural Ensemble Classifier for Semantic Relation Extraction in Scientific Papers Kyeongmin Rim, Jingxuan Tu, Kelley Lynch, James Pustejovsky Brandeis University, Department of Computer Science {krim,jxtu,kmlynch,jamesp}@brandeis.edu Abstract Within the natural language processing (NLP) community, shared tasks play an important role. They define common goals and allow the comparison of different methods on the shared problems and data. SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers (SE18T7) involves the identification and classification of relations in abstracts from computational linguistics (CL) publications. In this paper we describe an attempt to reproduce the methods and results from the top performing system at for SE18T7. We describe challenges we encountered in the process, report on the results of our system, and discuss the ways that our attempt at reproduction can inform best practices. Keywords: REPROLANG, Reproducibility, Semantic Relation Extraction, Scientific Information Extraction 1. Introduction Replicability and reproducibility are core ideas of modern scientific methods (Ivie and Thain, 2018), and thus repro- ducibility of scientific research results is an important topic in many research areas. Particularly over the last decade, terms such as “reproducibility crisis” or “p-hacking” have gained attention from different scientific communities, es- pecially after studies showed failure in reproduction of re- searches in life and medicine science (Begley and Ellis, 2012), psychology ans behavioral science (Anderson et al., 2015), as well as astrophysics (Collaboration and others, 2015). The same studies also showed that a complete and successful replication of experiments and results in some published articles with huge findings and impact is ex- tremely difficult or nearly impossible because of the lack of model details and experiment settings . The field of natural language processing (NLP) and com- putational linguistics (CL) hasn’t been an exception. The questions surrounding reproducibility of published work have been getting researchers attention over the last decade. (Fokkens et al., 2013; Hagen et al., 2015). More recently, Language Resource and Evaluation initiated a proposal to establish a venue for publications on the topic of replicabil- ity and reproducibility to encourage researchers to actively investigate existing methods and experiments, and discuss the positive or negative results (Branco et al., 2017). Work- shops dedicated to these topics include • RRNLP 2015 - Workshop on Replicability and Reusability in Natural Language Processing1 • 4REAL Workshop - Workshop on Research Results Re- producibility and Resources Citation in Science and Technology of Language2 (Branco et al., 2016) • 4REAL 2018 - Workshop on Research Results Repro- ducibility and Resources Citation in Science and Tech- nology of Language3 (Branco et al., 2018) , that aimed to address the need for replication of published results and push the discussion of this topic much further. A 1http://nl.ijs.si/rrnlp2015/ 2http://4real.di.fc.ul.pt/ 3http://4real2018.di.fc.ul.pt/ recent study (Mieskes et al., 2019) showed, by conducting a survey, the significance of reproducibility to a scientific community as well as the importance of having …

Content truncated to 500 words