Offspring from Reproduction Problems: What Replication Failure Teaches Us

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1691–1701, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Offspring from Reproduction Problems: What Replication Failure Teaches Us Antske Fokkens and Marieke van Erp The Network Institute VU University Amsterdam Amsterdam, The Netherlands {a.s.fokkens,m.g.j.van.erp}@vu.nl Marten Postma Utrecht University Utrecht, The Netherlands martenp@gmail.com Ted Pedersen Dept. of Computer Science University of Minnesota Duluth, MN 55812 USA tpederse@d.umn.edu Piek Vossen The Network Institute VU University Amsterdam Amsterdam, The Netherlands piek.vossen@vu.nl Nuno Freire The European Library The Hague, The Netherlands nfreire@gmail.com Abstract Repeating experiments is an important in- strument in the scientific toolbox to vali- date previous work and build upon exist- ing work. We present two concrete use cases involving key techniques in the NLP domain for which we show that reproduc- ing results is still difficult. We show that the deviation that can be found in repro- duction efforts leads to questions about how our results should be interpreted. Moreover, investigating these deviations provides new insights and a deeper under- standing of the examined techniques. We identify five aspects that can influence the outcomes of experiments that are typically not addressed in research papers. Our use cases show that these aspects may change the answer to research questions leading us to conclude that more care should be taken in interpreting our results and more research involving systematic testing of methods is required in our field. 1 Introduction Research is a collaborative effort to increase knowledge. While it includes validating previous approaches, our experience is that most research output in our field focuses on presenting new ap- proaches, and to a somewhat lesser extent building upon existing work. In this paper, we argue that the value of research that attempts to replicate previous approaches goes beyond simply validating what is already known. It is also an essential aspect for building upon existing approaches. Especially when validation fails or variations in results are found, systematic testing helps to obtain a clearer picture of both the approach itself and of the meaning of state-of-the- art results leading to a better insight into the qual- ity of new approaches in relation to previous work. We support our claims by presenting two use cases that aim to reproduce results of previous work in two key NLP technologies: measuring WordNet similarity and Named Entity Recogni- tion (NER). Besides highlighting the difficulty of repeating other researchers’ work, new insights about the approaches emerged that were not pre- sented in the original papers. This last point shows that reproducing results is not merely part of good practice in science, but also an essential part in gaining a better understanding of the methods we use. Likewise, the problems we face in reproduc- ing previous results are not merely frustrating in- conveniences, but also pointers to research ques- tions that deserve deeper investigation. We investigated five aspects that cause exper- imental variation that are not typically described in publications: preprocessing (e.g. tokenisa- tion), experimental setup (e.g. …

Content truncated to 500 words