Reproducible Results in Parsing-Based Machine Translation: The JHU Shared Task Submission

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 177–182, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Reproducible Results in Parsing-Based Machine Translation: The JHU Shared Task Submission Lane Schwartz ∗ University of Minnesota Minneapolis, MN lane@cs.umn.edu Abstract We present the Johns Hopkins Univer- sity submission to the 2010 WMT shared translation task. We describe processing steps using open data and open source software used in our submission, and pro- vide the scripts and configurations re- quired to train, tune, and test our machine translation system. 1 Introduction Research investigating natural language process- ing and computational linguistics can and should have an extremely low barrier to entry. The data with which we work is customarily available in common electronic formats. The computational techniques which we apply can typically be per- formed on commodity computing resources which are widely available. In short, there should be no reason why small research groups and even lone researchers should not be able to join and make substantive contributions furthering our field. The reality is less encouraging. Many published articles describe novel tech- niques and provide interesting results, yet fail to describe technical details in sufficient detail to al- low their results to be reproduced by other re- searchers. While there are notable and laudable exceptions, many publications fail to provide the source code and scripts necessary to reproduce re- sults. The use of restricted data, not freely avail- able for download by any interested researcher only compounds these problems. Pedersen (2008) rightly argues that the implementation details so often ignored in publications are in fact essential for our research to be reproducible science. Reproducibility in machine translation is made more challenging by the complexity of experi- mental workflows. Results in machine translation ∗Research conducted as a visiting researcher at Johns Hopkins University tasks are dependent on a cascade of processing steps and configurations. While interesting sub- sets of these usually appear in experimental de- scriptions, many steps (preprocessing techniques, alignment parameters, translation rule extraction parameters, language model parameters, list of features used) are invariably omitted, even though these configurations are often critical to reproduc- ing results. This paper describes the Johns Hopkins Univer- sity submission to the 2010 Workshop on Statis- tical Machine Translation shared translation task. Links to the software, scripts, and configurations used to run the experiments described herein are provided. The remainder of this paper is struc- tured as follows. Section 2 lists the major ex- amples of publicly available open source machine translation systems, parallel corpora, and machine translation workflow management systems. Sec- tion 3 describes the experimental workflow used to run the shared task translations, with the corre- sponding experimental design in section 4. Sec- tion 5 presents the shared task results. 2 Related Work The last four years have witnessed the implemen- tation and release of numerous open source ma- chine translation systems. The widely used Moses system (Koehn et al., 2007) implements the stan- dard phrase-based translation model. Parsing- based …

Content truncated to 500 words