Joshua 2.0: A Toolkit for Parsing-Based Machine Translation with Syntax, Semirings, Discriminative Training and Other Goodies

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 133–137, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Joshua 2.0: A Toolkit for Parsing-Based Machine Translation with Syntax, Semirings, Discriminative Training and Other Goodies Zhifei Li, Chris Callison-Burch, Chris Dyer,† Juri Ganitkevitch, Ann Irvine, Sanjeev Khudanpur, Lane Schwartz,⋆ Wren N. G. Thornton, Ziyuan Wang, Jonathan Weese and Omar F. Zaidan Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD † Computational Linguistics and Information Processing Lab, University of Maryland, College Park, MD ⋆ Natural Language Processing Lab, University of Minnesota, Minneapolis, MN Abstract We describe the progress we have made in the past year on Joshua (Li et al., 2009a), an open source toolkit for parsing based machine translation. The new functional- ity includes: support for translation gram- mars with a rich set of syntactic nonter- minals, the ability for external modules to posit constraints on how spans in the in- put sentence should be translated, lattice parsing for dealing with input uncertainty, a semiring framework that provides a uni- fied way of doing various dynamic pro- gramming calculations, variational decod- ing for approximating the intractable MAP decoding, hypergraph-based discrimina- tive training for better feature engineering, a parallelized MERT module, document- level and tail-based MERT, visualization of the derivation trees, and a cleaner pipeline for MT experiments. 1 Introduction Joshua is an open-source toolkit for parsing-based machine translation that is written in Java. The initial release of Joshua (Li et al., 2009a) was a re-implementation of the Hiero system (Chiang, 2007) and all its associated algorithms, includ- ing: chart parsing, n-gram language model inte- gration, beam and cube pruning, and k-best ex- traction. The Joshua 1.0 release also included re-implementations of suffix array grammar ex- traction (Lopez, 2007; Schwartz and Callison- Burch, 2010) and minimum error rate training (Och, 2003; Zaidan, 2009). Additionally, it in- cluded parallel and distributed computing tech- niques for scalability (Li and Khudanpur, 2008). This paper describes the additions to the toolkit over the past year, which together form the 2.0 re- lease. The software has been heavily used by the authors and several other groups in their daily re- search, and has been substantially refined since the first release. The most important new functions in the toolkit are: • Support for any style of synchronous context free grammar (SCFG) including syntax aug- ment machine translation (SAMT) grammars (Zollmann and Venugopal, 2006) • Support for external modules to posit transla- tions for spans in the input sentence that con- strain decoding (Irvine et al., 2010) • Lattice parsing for dealing with input un- certainty, including ambiguous output from speech recognizers or Chinese word seg- menters (Dyer et al., 2008) • A semiring architecture over hypergraphs that allows many inference operations to be implemented easily and elegantly (Li and Eisner, 2009) • Improvements to decoding through varia- tional decoding and other approximate meth- ods that overcome intractable MAP decoding (Li et al., 2009b) • Hypergraph-based discriminative training for better …

Content truncated to 500 words