zymake: A Computational Workflow System for Machine Learning and Natural Language Processing

Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 5–13, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics zymake: a computational workflow system for machine learning and natural language processing Eric Breck Department of Computer Science Cornell University Ithaca, NY 14853 USA ebreck@cs.cornell.edu Abstract Experiments in natural language processing and machine learning typically involve run- ning a complicated network of programs to create, process, and evaluate data. Re- searchers often write one or more UNIX shell scripts to “glue” together these various pieces, but such scripts are suboptimal for several rea- sons. Without significant additional work, a script does not handle recovering from fail- ures, it requires keeping track of complicated filenames, and it does not support running pro- cesses in parallel. In this paper, we present zymake as a solution to all these problems. zymake scripts look like shell scripts, but have semantics similar to makefiles. Using zymake improves repeatability and scalabil- ity of running experiments, and provides a clean, simple interface for assembling compo- nents. A zymake script also serves as doc- umentation for the complete workflow. We present a zymake script for a published set of NLP experiments, and demonstrate that it is superior to alternative solutions, including shell scripts and makefiles, while being far simpler to use than scientific grid computing systems. 1 Introduction Running experiments in natural language process- ing and machine learning typically involves a com- plicated network of programs. One program might extract data from a raw corpus, others might pre- process it with various linguistic tools, before finally the main program being tested is run. Further pro- grams must evaluate the output, and produce graphs and tables for inclusion in papers and presentations. All of these steps can be run by hand, but a more typ- ical approach is to automate them using tools such as UNIX shell scripts. We argue that any approach should satisfy a number of basic criteria. Reproducibility At some future time, the original researcher or other researchers ought to be able to re-run the set of experiments and produce identical results1. Such reproducibility is a cornerstone of sci- entific research, and ought in principle to be easier in our discipline than in a field requiring physical measurements such as physics or chemistry. Simplicity We want to create a system that we and other researchers will find easy to use. A system which requires significant overhead before any ex- periment can be run can limit a researcher’s ability to quickly and easily try out new ideas. A realistic life-cycle of experiments A typical ex- periment evolves in structure as it goes along - the researcher may choose partway through to add new datasets, new ranges of parameters, or new sets of models to test. Moreover, a computational exper- iment rarely works correctly the first time. Com- ponents break for various reasons, a tool may not perform as expected, and so forth. A usable tool must be simple to use in the face of such …

Content truncated to 500 words