Joshua 5.0: Sparser, Better, Faster, Server

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 206–212, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Joshua 5.0: Sparser, better, faster, server Matt Post1 and Juri Ganitkevitch2 and Luke Orland1 and Jonathan Weese2 and Yuan Cao2 1Human Language Technology Center of Excellence 2Center for Language and Speech Processing Johns Hopkins University Chris Callison-Burch Computer and Information Sciences Department University of Pennsylvania Abstract We describe improvements made over the past year to Joshua, an open-source trans- lation system for parsing-based machine translation. The main contributions this past year are significant improvements in both speed and usability of the grammar extraction and decoding steps. We have also rewritten the decoder to use a sparse feature representation, enabling training of large numbers of features with discrimina- tive training methods. 1 Introduction Joshua is an open-source toolkit1 for hierarchical and syntax-based statistical machine translation of human languages with synchronous context- free grammars (SCFGs). The original version of Joshua (Li et al., 2009) was a port (from Python to Java) of the Hiero machine translation system in- troduced by Chiang (2007). It was later extended to support grammars with rich syntactic labels (Li et al., 2010). Subsequent efforts produced Thrax, the extensible Hadoop-based extraction tool for synchronous context-free grammars (Weese et al., 2011), later extended to support pivoting-based paraphrase extraction (Ganitkevitch et al., 2012). Joshua 5.0 continues our yearly update cycle. The major components of Joshua 5.0 are: §3.1 Sparse features. Joshua now supports an easily-extensible sparse feature implementa- tion, along with tuning methods (PRO and kbMIRA) for efficiently setting the weights on large feature vectors. 1joshua-decoder.org §3.2 Significant speed increases. Joshua 5.0 is up to six times faster than Joshua 4.0, and also does well against hierarchical Moses, where end-to-end decoding (including model load- ing) of WMT test sets is as much as three times faster. §3.3 Thrax 2.0. Our reengineered Hadoop-based grammar extractor, Thrax, is up to 300% faster while using significantly less interme- diate disk space. §3.4 Many other features. Joshua now includes a server mode with fair round-robin scheduling among and within requests, a bundler for dis- tributing trained models, improvements to the Joshua pipeline (for managing end-to-end ex- periments), and better documentation. 2 Overview Joshua is an end-to-end statistical machine trans- lation toolkit. In addition to the decoder com- ponent (which performs the actual translation), it includes the infrastructure needed to prepare and align training data, build translation and language models, and tune and evaluate them. This section provides a brief overview of the contents and abilities of this toolkit. More infor- mation can be found in the online documentation (joshua-decoder.org/5.0/). 2.1 The Pipeline: Gluing it all together The Joshua pipeline ties together all the infrastruc- ture needed to train and evaluate machine transla- tion systems for research or industrial purposes. Once data has been segmented into parallel train- ing, development, and test sets, a single invocation of the pipeline script is enough to invoke this entire infrastructure from beginning to end. Each step is …

Content truncated to 500 words