Reproduction and Revival of the Argument Reasoning Comprehension Task

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5055–5064 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5055 Reproduction and Revival of the Argument Reasoning Comprehension Task Jo˜ao Rodrigues, Ruben Branco, Jo˜ao Silva, Ant´onio Branco University of Lisbon NLX—Natural Language and Speech Group, Department of Informatics Faculdade de Ciˆencias Campo Grande, 1749-016 Lisboa, Portugal {joao.rodrigues, ruben.branco, jsilva, antonio.branco}@di.fc.ul.pt Abstract Reproduction of scientific findings is essential for scientific development across all scientific disciplines and reproducing results of previous works is a basic requirement for validating the hypothesis and conclusions put forward by them. This paper reports on the scientific reproduction of several systems addressing the Argument Reasoning Comprehension Task of SemEval2018. Given a recent publication that pointed out spurious statistical cues in the data set used in the shared task, and that produced a revised version of it, we also evaluated the reproduced systems with this new data set. The exercise reported here shows that, in general, the reproduction of these systems is successful with scores in line with those reported in SemEval2018. However, the performance scores are worst than those, and even below the random baseline, when the reproduced systems are run over the revised data set expunged from data artifacts. This demonstrates that this task is actually a much harder challenge than what could have been perceived from the inflated, close to human-level performance scores obtained with the data set used in SemEval2018. This calls for a revival of this task as there is much room for improvement until systems may come close to the upper bound provided by human performance. Keywords: Reproduction, Replication, Argument Mining, Argument Reasoning, Argument Comprehension, ARCT, SemEval. 1. Introduction The ability to repeat experiments and to reproduce their re- sults is a cornerstone of scientific work and it is necessary to properly validate the findings that are published. Sev- eral examples of failed replication efforts, however, have been documented in different scientific fields (Branco et al., 2017). For Natural Language Processing (NLP), failure to reproduce results has been reported for WordNet similarity measures (Fokkens et al., 2013), sentiment analysis (Moore and Rayson, 2018), PoS tagging and named entity recogni- tion (Reimers and Gurevych, 2017), among others. The importance of open challenges that foster the reproduc- tion of research results has been underlined (Fokkens et al., 2013; Branco, 2013) as part of the solution for the so-called replication crisis (Hutson, 2018). The community gath- ered around the science and technology of language has re- sponded through initiatives such as the 4REAL workshops (Branco et al., 2017; Branco et al., 2018), and the forthcom- ing ReproLang cooperative shared task (ReproLang, 2019) of the LREC2020 conference. The discussion regarding the reproduction of scientific ex- periments faces an additional, non-technical difficulty as there is no established consensus on the terminology that should be used, with the terms replicability, reproducibility and repeatability being used interchangeably in the litera- ture. Here we adopt the definitions given by Stodden et al. …

Content truncated to 500 words