Community Perspective on Replicability in Natural Language Processing

Proceedings of Recent Advances in Natural Language Processing, pages 768–775, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_089 768 NLP Community Perspectives on Replicability Margot Mieskes University of Applied Sciences, Darmstadt Germany margot.mieskes@h-da.de Kar¨en Fort Sorbonne Universit´e, EA STIH Paris, France karen.fort@sorbonne-universite.fr Aur´elie N´ev´eol LIMSI, CNRS, Universit´e Paris-Saclay France neveol@limsi.fr Cyril Grouin LIMSI, CNRS, Universit´e Paris-Saclay France cyril.grouin@limsi.fr Kevin Cohen Computational Bioscience Program University of Colorado, USA kevin.cohen@gmail.com Abstract With recent efforts in drawing attention to the task of replicating and/or reproduc- ing1 results, for example in the context of COLING 2018 and various LREC work- shops, the question arises how the NLP community views the topic of replicabil- ity in general. Using a survey, in which we involve members of the NLP commu- nity, we investigate how our community perceives this topic, its relevance and op- tions for improvement. Based on over t- wo hundred participants, the survey results confirm earlier observations, that success- ful reproducibility requires more than hav- ing access to code and data. Additionally, the results show that the topic has to be tackled from the authors’, reviewers’ and community’s side. 1 Introduction “As a community, we need to know where our approaches fail, as much – if not more so – as where they succeed.” Despite this statement by Fokkens et al. (2013), we are still aiming at high- er, faster, better results with little outside verifica- tion. And although it has become good practise to share code, data and parameters, previous work and experience indicate that sharing is still not as common as one would hope for. Call for Papers in major conferences encourage to submit or refer- ence data and to submit code, i.e., in supplemental material2. But recent work indicates that this is 1We use replication to describe related efforts, regardless of the exact aim (see (Cohen et al., 2018)) 2see for example http://www.acl2019.org/EN/ call-for-papers.xhtml “ACL (. . .) encourages the submission of supplementary material to report (. . .) details necessary for the replication of the experiments.” not done thoroughly enough (Mieskes, 2017; Co- hen et al., 2017; Wieling et al., 2018). Other fac- tors mentioned by Fokkens et al. (2013), such as preprocessing methods, experimental setup, sys- tem variation, etc., are rarely reported. Pedersen (2008) urges to not fear any dispossession of a tool, and highlights that sharing code will result in its use in new systems and citations of the work describing this code. Moreover, we should con- sider sharing software as a way to improve it more efficiently. In recent years, we have seen a rise in attention targeted towards the task of replica- tion for example in the COLING 2018 selection criteria3, the LREC 4REAL Workshops (Branco et al., 2016, 2018) and the recent LREC initiative for replication.4 But so far the task of replicat- ing previous results has little merit in itself, but is rather only a (baseline) part in a paper. Addition- ally, normally it only gets reported in successful or mainly successful cases. Following the argument …

Content truncated to 500 words