A Shared Task of a New, Collaborative Type to Foster Reproducibility: A First Exercise in the Area of Language Science and Technology with REPROLANG2020

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5539–5545 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5539 A Shared Task of a New, Collaborative Type to foster Reproducibility: A first exercise in the area of language science and technology with REPROLANG2020 Ant´onio Branco,1 Nicoletta Calzolari,2 Piek Vossen,3 Gertjan van Noord,4 Dieter Van Uytvank,5 Jo˜ao Silva,1 Lu´ıs Gomes,1 Andr´e Moreira,5 Willem Elbers5 1University of Lisbon, Department of Informatics, Faculdade de Ciˆencias, Portugal, antonio.branco@di.fc.ul.pt 2Istituto di Linguistica Computazionale, CNR, Pisa, glottolo@ilc.cnr.it 3Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands, piek.vossen@vu.nl 4University of Groningen, Groningen, The Netherlands, g.j.m.van.noord@rug.nl 5CLARIN ERIC, Utrecht, The Netherlands, firstname@clarin.eu Abstract In this paper, we introduce a new type of shared task — which is collaborative rather than competitive — designed to support and foster the reproduction of research results. We also describe the first event running such a novel challenge, present the results obtained, discuss the lessons learned and ponder on future undertakings. Keywords: reproduction, replication, natural language processing, computational linguistics, language technology 1. Introduction Scientific knowledge is grounded on falsifiable predictions and thus its credibility and raison d’ˆetre rely on the possi- bility of repeating experiments and getting similar results as originally obtained and reported. In many young scien- tific areas, including Natural Language Processing (NLP), acknowledgement and promotion of the reproduction of re- search results need to be increased (Branco, 2013). To raise awareness of the importance of reproducibility in NLP, we organised a community-wide shared task at LREC2020—The 12th International Conference on Lan- guage Resources and Evaluation—, to elicit and motivate the spread of scientific work on reproduction. This ini- tiative builds on the previous pioneer LREC workshops on reproducibility 4REAL2016 (Branco et al., 2016) and 4REAL2018 (Branco et al., 2018). It follows also the ini- tiative of the Language Resources and Evaluation journal, with its special section on reproducibility and replicability (Branco et al., 2017). Shared tasks are an important instrument to stimulate scien- tific research and to advance the state of the art in many ar- eas and topics in a measurable fashion. They facilitate com- petition among research teams that seek to resolve a com- mon problem or task with the best possible solution or per- formance. The proposed task is typically a well-described yet scientifically challenging problem and the submitted so- lutions by the different teams are evaluated against the same test sets for comparison (kept secret during the develop- ment phase). In this paper, we introduce a new type of shared task— which is collaborative rather than competitive—designed to support and foster the reproduction of research results: “the calculation of quantitative scientific results by independent scientists using the original data sets and methods”, (Stod- den et al., 2014, Preface, p. vii). We also describe the first event running such a novel chal- lenge, present the results obtained, discuss the lessons learned and ponder on future undertakings. The task, called REPROLANG-The Shared Task on the Reproduction of …

Content truncated to 500 words