Joshua: An Open Source Toolkit for Parsing-Based Machine Translation

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 135–139, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics Joshua: An Open Source Toolkit for Parsing-based Machine Translation Zhifei Li, Chris Callison-Burch, Chris Dyer,† Juri Ganitkevitch,+ Sanjeev Khudanpur, Lane Schwartz,⋆ Wren N. G. Thornton, Jonathan Weese and Omar F. Zaidan Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD † Computational Linguistics and Information Processing Lab, University of Maryland, College Park, MD + Human Language Technology and Pattern Recognition Group, RWTH Aachen University, Germany ⋆ Natural Language Processing Lab, University of Minnesota, Minneapolis, MN Abstract We describe Joshua, an open source toolkit for statistical machine transla- tion. Joshua implements all of the algo- rithms required for synchronous context free grammars (SCFGs): chart-parsing, n- gram language model integration, beam- and cube-pruning, and k-best extraction. The toolkit also implements suffix-array grammar extraction and minimum error rate training. It uses parallel and dis- tributed computing techniques for scala- bility. We demonstrate that the toolkit achieves state of the art translation per- formance on the WMT09 French-English translation task. 1 Introduction Large scale parsing-based statistical machine translation (e.g., Chiang (2007), Quirk et al. (2005), Galley et al. (2006), and Liu et al. (2006)) has made remarkable progress in the last few years. However, most of the systems mentioned above employ tailor-made, dedicated software that is not open source. This results in a high bar- rier to entry for other researchers, and makes ex- periments difficult to duplicate and compare. In this paper, we describe Joshua, a general-purpose open source toolkit for parsing-based machine translation, serving the same role as Moses (Koehn et al., 2007) does for regular phrase-based ma- chine translation. Our toolkit is written in Java and implements all the essential algorithms described in Chiang (2007): chart-parsing, n-gram language model in- tegration, beam- and cube-pruning, and k-best ex- traction. The toolkit also implements suffix-array grammar extraction (Lopez, 2007) and minimum error rate training (Och, 2003). Additionally, par- allel and distributed computing techniques are ex- ploited to make it scalable (Li and Khudanpur, 2008b). We have also made great effort to ensure that our toolkit is easy to use and to extend. The toolkit has been used to translate roughly a million sentences in a parallel corpus for large- scale discriminative training experiments (Li and Khudanpur, 2008a). We hope the release of the toolkit will greatly contribute the progress of the syntax-based machine translation research.1 2 Joshua Toolkit When designing our toolkit, we applied general principles of software engineering to achieve three major goals: Extensibility, end-to-end coherence, and scalability. Extensibility: The Joshua code is organized into separate packages for each major aspect of functionality. In this way it is clear which files contribute to a given functionality and researchers can focus on a single package without worrying about the rest of the system. Moreover, to mini- mize the problems of unintended interactions and unseen dependencies, which is common hinder- ance to extensibility in large projects, all …

Content truncated to 500 words