Preserving Workflow Reproducibility: The RePlay-DH Client as a Tool for Process Documentation

Preserving Workflow Reproducibility: The RePlay-DH Client as a Tool for Process Documentation Markus G¨artner1* Uli Hahn2** Sibylle Hermann3* 1Institute for Natural Language Processing 2Communication and Information Centre 3University Library *University of Stuttgart **Ulm University markus.gaertner@ims.uni-stuttgart.de, uli.hahn@uni-ulm.de sibylle.hermann@ub.uni-stuttgart.de Abstract In this paper we present a software tool for elicitation and management of process metadata. It follows our previously published design idea of an assistant for researchers that aims at minimizing the additional effort required for producing a sustainable workflow documentation. With the ever-growing number of linguistic resources available, it also becomes increasingly important to provide proper documentation to make them comparable and to allow meaningful evaluations for specific use cases. The often prevailing practice of post hoc documentation of resource generation or research processes bears the risk of information loss. Not only does detailed documentation of a process aid in achieving reproducibility, it also increases usefulness of the documented work for others as a cornerstone of good scientific practice. Time pressure together with the lack of simple documentation methods leads to workflow documentation in practice being an arduous and often neglected task. Our tool ensures a clean documentation for common workflows in natural language processing and digital humanities. Additionally, it can easily be integrated into existing institutional infrastructures. Keywords: reproducibility, metadata, documentation 1. Introduction Several decades ago Claerbout and Karrenbach (1992) for- mulated the idea that in a research environment of ever in- creasing digitization the published articles are only adver- tisement for the actual scholarship. The scholarship itself then also includes all the scripts, tools, data and additional information that was used to create the results. This view is even more pronounced today, considering how already mi- nor adjustments or changes in intermediary or preprocess- ing steps in typical natural language processing (NLP) tasks can have significant impact on the outcome of down-stream tasks, as has been exemplified by Elming et al. (2013). While it is not an ultimate guarantee for reproducibility, thorough documentation of workflows certainly aids in be- ing able to reproduce results, evaluate the suitability of re- sources for a certain task or at least increases ite)usability for others and even for oneself. In stark contrast to this, day-to-day research often exhibits a serious neglect of documentation efforts when it comes to minor details in a project’s workflow. Reasons for this include for example the competitive and time pressure that largely dominates today’s research practice. On the road to a publishable end-result it is easy to omit workflow docu- mentation, especially if it requires a substantial amount of extra effort and there is no apparent gratification for it. If at all, such documentation is typically created retrospectively at a later time, where the risk that some details of the work- flow might have already been lost is high. The matter is further complicated by the diverse nature of workflows in the field of NLP, computational linguistics (CL) or digital humanities (DH). They draw from a vast pool of available resources and tools to intermix strictly au- tomatized …

Content truncated to 500 words