Three Dimensions of Reproducibility in Natural Language Processing

Three Dimensions of Reproducibility in Natural Language Processing K. Bretonnel Cohen1,2, Jingbo Xia3, Pierre Zweigenbaum2, Tiffany J. Callahan1, Orin Hargraves4, Foster Goss5, Nancy Ide6, Aur´elie N´ev´eol2, Cyril Grouin2, and Lawrence E. Hunter1 1Computational Bioscience Program, University of Colorado School of Medicine 2LIMSI, CNRS, Universit´e Paris-Saclay 3Huazhong Agricultural University 4University of Colorado Boulder 5Department of Emergency Medicine, University of Colorado 6Vassar College Abstract Despite considerable recent attention to problems with reproducibility of scientific research, there is a striking lack of agreement about the definition of the term. That is a problem, because the lack of a consensus definition makes it difficult to compare studies of reproducibility, and thus to have even a broad overview of the state of the issue in natural language processing. This paper proposes an ontology of reproducibility in that field. Its goal is to enhance both future research and communication about the topic, and retrospective meta-analyses. We show that three dimensions of reproducibility, corresponding to three kinds of claims in natural language processing papers, can account for a variety of types of research reports. These dimensions are reproducibility of a conclusion, of a finding, and of a value. Three biomedical natural language processing papers by the authors of this paper are analyzed with respect to these dimensions. Keywords: methodology, reproducibility, repeatability, replicability, replicatability 1. Introduction The journal Language Resources and Evaluation recently published an editorial on reproducibility in language pro- cessing. The editorial announced a new topic for the jour- nal, named the associate editors for the topic, called for pa- pers on the topic, and defined a number of relevant terms (Branco et al., 2017). Before the editorial had appeared, the authors (who include two of the authors of this paper) had already submitted a correction: on further assessment of the literature, they had realized that they should reverse the definitions of two cru- cial terms (viz. reproducibility and replicability)1. It was a textbook example of publication of an analysis that turned out not to hold, and of how such an analysis should be han- dled. It is especially striking that the topic of the correction was the definition of two rather central terms—failures of repro- ducibility have been so present in both the global scientific and the public consciousness that one could reasonably ex- pect that there would be at least a broad consensus about the terminology that is used to discuss the phenomenon. And yet: even a cursory (and certainly a careful) review of the literature shows that no such consensus exists. Indeed, the community is not even close to a consensus. In defini- tions in the literature (multiple examples omitted from the abstract due to space constraints), one observes at least the following three terms used frequently to refer to the same two concerns: replicability, repeatability, and reproducibil- ity. Less frequently, one also sees the terms commensurate, valid, and validity, and all of these can be involved in the definition of rigor (Kilicoglu, 2017). It is not uncommon to see reproducibility and replicability …

Content truncated to 500 words