Reproduction and Replication: A Case Study with Automatic Essay Scoring

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5603–5613 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5603 Reproduction and Replication: A Case Study with Automatic Essay Scoring Eva Huber, Ça˘grı Çöltekin University of Tübingen eva.huber@student.uni-tuebingen.de, ccoltekin@sfs.uni-tuebingen.de Abstract As in many experimental sciences, reproducibility of experiments has gained ever more attention in the NLP community. This paper presents our reproduction efforts of an earlier study of automatic essay scoring (AES) for determining the proficiency of second language learners in a multilingual setting. We present three sets of experiments with different objectives. First, as prescribed by the LREC 2020 REPROLANG shared task, we rerun the original AES system using the code published by the original authors on the same dataset. Second, we repeat the same experiments on the same data with a different implementation. And third, we test the original system on a different dataset and a different language. Most of our findings are in line with the findings of the original paper. Nevertheless, there are some discrepancies between our results and the results presented in the original paper. We report and discuss these differences in detail. We further go into some points related to confirmation of research findings through reproduction, including the choice of the dataset, reporting and accounting for variability, use of appropriate evaluation metrics, and making code and data available. We also discuss the varying uses and differences between the terms reproduction and replication, and we argue that reproduction, the confirmation of conclusions through independent experiments in varied settings is more valuable than exact replication of the published values. 1. Introduction Confirmation of results through independent replication is a well-established practice in experimental sciences. How- ever, a number of recent negative reproduction results in medical and behavioural sciences indicate that a significant number of published results are not verifiable through inde- pendent reproduction efforts (Open Science Collaboration, 2015; Freedman et al., 2015, for example). The issue, of- ten titled reproducibility crisis, has become influential both in scientific communities and popular media. Similar con- cerns have been raised in the fields of machine learning and artificial intelligence (Kitzes et al., 2017; Hutson, 2018; Raff, 2019). As a field that heavily relies on experimental work, the same concerns apply to computational linguis- tics and natural language processing (NLP), and there have been recent efforts to understand the extent of the problem and identify potential solutions. The present work is con- ducted in the context of such an effort, REPROLANG 2020 shared task on reproducibility of results in computational studies of language.1 Our study, in particular, is concerned with the reproduc- tion of a study of automatic assay scoring (AES) for de- termining language proficiency levels of second language learners (Vajjala and Rama, 2018). In AES, the aim is to assign a score, mark or level to a text by means of an au- tomatic system. The motivation behind the development of such systems lies in the time, cost and reliability …

Content truncated to 500 words