Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking

Findings of the Association for Computational Linguistics: ACL 2022, pages 3864 - 3876 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking Tianyi Luo1, Rui Meng2, Xin Eric Wang1, Yang Liu1 1Computer Science and Engineering, University of California, Santa Cruz 2Lawrence Berkeley National Laboratory, University of California, Berkeley {tluo6, xwang366, yangliu}@ucsc.edu rmeng@lbl.gov Abstract Research Replication Prediction (RRP) is the task of predicting whether a published research result can be replicated or not. Building an interpretable neural text classifier for RRP pro- motes the understanding of why a research pa- per is predicted as replicable or non-replicable and therefore makes its real-world application more reliable and trustworthy. However, the prior works on model interpretation mainly fo- cused on improving the model interpretabil- ity at the word/phrase level, which are insuf- ficient especially for long research papers in RRP. Furthermore, the existing methods can- not utilize a large size of unlabeled dataset to further improve the model interpretabil- ity. To address these limitations, we aim to build an interpretable neural model which can provide sentence-level explanations and apply weakly supervised approach to further lever- age the large corpus of unlabeled datasets to boost the interpretability in addition to improv- ing prediction performance as existing works have done. In this work, we propose the Variational Contextual Consistency Sentence Masking (VCCSM) method to automatically extract key sentences based on the context in the classifier, using both labeled and unla- beled datasets. Results of our experiments on RRP along with European Convention of Hu- man Rights (ECHR) datasets demonstrate that VCCSM is able to improve the model inter- pretability for the long document classification tasks using the area over the perturbation curve and post-hoc accuracy as evaluation metrics. 1 Introduction Scientific research results that cannot be repro- duced are unreliable and negatively impact the de- velopment of science. Therefore, it is important to know whether a published research result can be replicated or not. To this end, domain researchers have conducted several direct replication projects in contemporary published social science studies The mean error rates appear in the lower part of Table 6. The pattern of errors was similar to that of the latency results. The error rate was not significantly higher for sum (27%) than for neutral targets (p=.159). Replicable or not Research Replication Prediction The mean error rates appear in the lower part of Table 6. The pattern of errors was similar to that of the latency results. The error rate was not significantly higher for sum (27%) than for neutral targets (p=.159). Our VCCSM Model Mask the first & second sentence Binary classification Sentence Masking Text Extraction Research Paper (pdf) (a) (b) Figure 1: (a) Given the text information of a research paper, Research Replication Prediction (RRP) task pre- dicts whether the paper can be reproduced or not. (b) Having the same input as (a), our VCCSM model can keep the important sentences (through masking unim- portant ones) which are related to reproducibility. …

Content truncated to 500 words