Joshua 3.0: Syntax-based Machine Translation with the Thrax Grammar Extractor

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 478–484, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Joshua 3.0: Syntax-based Machine Translation with the Thrax Grammar Extractor Jonathan Weese1, Juri Ganitkevitch1, Chris Callison-Burch1, Matt Post2 and Adam Lopez1,2 1Center for Language and Speech Processing 2Human Language Technology Center of Excellence Johns Hopkins University Abstract We present progress on Joshua, an open- source decoder for hierarchical and syntax- based machine translation. The main fo- cus is describing Thrax, a flexible, open source synchronous context-free grammar ex- tractor. Thrax extracts both hierarchical (Chi- ang, 2007) and syntax-augmented machine translation (Zollmann and Venugopal, 2006) grammars. It is built on Apache Hadoop for efficient distributed performance, and can eas- ily be extended with support for new gram- mars, feature functions, and output formats. 1 Introduction Joshua is an open-source1 toolkit for hierarchical machine translation of human languages. The origi- nal version of Joshua (Li et al., 2009) was a reim- plementation of the Python-based Hiero machine- translation system (Chiang, 2007); it was later ex- tended (Li et al., 2010) to support richer formalisms, such as SAMT (Zollmann and Venugopal, 2006). The main focus of this paper is to describe this past year’s work in developing Thrax (Weese, 2011), an open-source grammar extractor for Hiero and SAMT grammars. Grammar extraction has shown itself to be something of a black art, with decod- ing performance depending crucially on a variety of features and options that are not always clearly described in papers. This hindered direct com- parison both between and within grammatical for- malisms. Thrax standardizes Joshua’s grammar ex- 1http://github.com/joshua-decoder/joshua traction procedures by providing a flexible and con- figurable means of specifying these settings. Sec- tion 3 presents a systematic comparison of the two grammars using identical feature sets. In addition, Joshua now includes a single pa- rameterized script that implements the entire MT pipeline, from data preparation to evaluation. This script is built on top of a module called CachePipe. CachePipe is a simple wrapper around shell com- mands that uses SHA-1 hashes and explicitly- provided lists of dependencies to determine whether a command needs to be run, saving time both in run- ning and debugging machine translation pipelines. 2 Thrax: grammar extraction In modern machine translation systems such as Joshua (Li et al., 2009) and cdec (Dyer et al., 2010), a translation model is represented as a synchronous context-free grammar (SCFG). Formally, an SCFG may be considered as a tuple (N, S, Tσ, Tτ, G) where N is a set of nonterminal symbols of the grammar, S ∈ N is the goal symbol, Tσ and Tτ are the source- and target-side terminal symbol vo- cabularies, respectively, and G is a set of production rules of the grammar. Each rule in G is of the form X → ⟨α, γ, ∼⟩ where X ∈ N is a nonterminal symbol, α is a se- quence of symbols from N ∪ Tσ, γ is a sequence of 478 symbols from N ∪ …

Content truncated to 500 words