Jane: Open Source Hierarchical Translation, Extended with Reordering and Lexicon Models

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 262–270, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Jane: Open Source Hierarchical Translation, Extended with Reordering and Lexicon Models David Vilar, Daniel Stein, Matthias Huck and Hermann Ney Lehrstuhl f¨ur Informatik 6 RWTH Aachen University Aachen, Germany {vilar,stein,huck,ney}@cs.rwth-aachen.de Abstract We present Jane, RWTH’s hierarchical phrase-based translation system, which has been open sourced for the scientific community. This system has been in de- velopment at RWTH for the last two years and has been successfully applied in dif- ferent machine translation evaluations. It includes extensions to the hierarchical ap- proach developed by RWTH as well as other research institutions. In this paper we give an overview of its main features. We also introduce a novel reordering model for the hierarchical phrase-based approach which further enhances transla- tion performance, and analyze the effect some recent extended lexicon models have on the performance of the system. 1 Introduction We present a new open source toolkit for hi- erarchical phrase-based translation, as described in (Chiang, 2007). The hierarchical phrase model is an extension of the standard phrase model, where the phrases are allowed to have “gaps”. In this way, long-distance dependencies and reorder- ings can be modelled in a consistent way. As in nearly all current statistical approaches to machine translation, this model is embedded in a log-linear model combination. RWTH has been developing this tool during the last two years and it was used success- fully in numerous machine translation evalua- tions. It is developed in C++ with special at- tention to clean code, extensibility and efficiency. The toolkit is available under an open source non-commercial license and downloadable from http://www.hltpr.rwth-aachen.de/jane. In this paper we give an overview of the main features of the toolkit and introduce two new ex- tensions to the hierarchical model. The first one is an additional reordering model inspired by the reordering widely used in phrase-based transla- tion systems and the second one comprises two extended lexicon models which further improve translation performance. 2 Related Work Jane implements many features presented in pre- vious work developed both at RWTH and other groups. As we go over the features of the system we will provide the corresponding references. Jane is not the first system of its kind, al- though it provides some unique features. There are other open source hierarchical decoders avail- able. These include • SAMT (Zollmann and Venugopal, 2006): The original version is not maintained any more and we had problems working on big corpora. A new version which requires Hadoop has just been released, however the documentation is still missing. • Joshua (Li et al., 2009): A decoder written in Java by the John Hopkins University. This project is the most similar to our own, how- ever both were developed independently and each one has some unique features. A brief comparison between these two systems is in- cluded in Section 5.1. • Moses (Koehn et al., 2007): The de-facto standard …

Content truncated to 500 words