Research Replication Prediction Using Weakly Supervised Learning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1464–1474 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1464 Research Replication Prediction Using Weakly Supervised Learning Tianyi Luo, Xingyu Li, Hainan Wang, Yang Liu Computer Science and Engineering, University of California, Santa Cruz, CA {tluo6, xli279, hwang207, yangliu}@ucsc.edu Abstract Knowing whether a published research result can be replicated is important. Carrying out di- rect replication of published research incurs a high cost. There are efforts tried to use ma- chine learning aided methods to predict sci- entific claims’ replicability. However, exist- ing machine learning aided approaches use only hand-extracted statistics features such as p-value, sample size, etc. without utilizing re- search papers’ text information and train only on a very small size of annotated data without making the most use of a large number of un- labeled articles. Therefore, it is desirable to develop effective machine learning aided au- tomatic methods which can automatically ex- tract text information as features so that we can benefit from Natural Language Processing techniques. Besides, we aim for an approach that benefits from both labeled and the large number of unlabeled data. In this paper, we propose two weakly supervised learning ap- proaches that use automatically extracted text information of research papers to improve the prediction accuracy of research replication us- ing both labeled and unlabeled datasets. Our experiments over real-world datasets show that our approaches obtain much better prediction performance compared to the supervised mod- els utilizing only statistic features and a small size of labeled dataset. Further, we are able to achieve an accuracy of 75.76% for predicting the replicability of research. 1 Introduction Non-reproducible scientific results will mislead the progress of science and undermine the trustworthi- ness of the research community. In recent years, we saw the emergence of systematic large-scale repli- cation projects which are based on the concerns of research credibility in the social and behavioral sci- ences (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014b, 2018; Collaboration et al., 2015). Researchers conducted preregistered repli- cations of hundreds of classic and contemporary published findings in the social and behavioral sci- ences. Unfortunately, the reported replication rates only range from 39% to 62%. Therefore it is im- portant to develop a confidence scoring system for the following question: To what extend can a research result be reproduced? The answer to the above question will help fa- cilitate the policymakers as well as the general public to better understand and digest a published claim. As a response, for example, Defense Ad- vanced Research Projects Agency (DARPA) has announced a systematic confidence checking of published claims (Russell, 2019). Alongside the above encouraging movement, the downside is that the average replication expense of each research project (which often consists a number of research studies) can go up to $500,000 (Freedman et al., 2015)1, which is hardly afford- able to replicate each research finding, with an exponentially increasing number of publications. Recently, efforts have been noted …

Content truncated to 500 words