Phramer - An Open Source Statistical Phrase-Based Translator

Proceedings of the Workshop on Statistical Machine Translation, pages 146–149, New York City, June 2006. c⃝2006 Association for Computational Linguistics Phramer - An Open Source Statistical Phrase-Based Translator Marian Olteanu, Chris Davis, Ionut Volosen and Dan Moldovan Human Language Technology Research Institute The University of Texas at Dallas Richardson, TX 75080 {marian,phoo,volosen,moldovan}@hlt.utdallas.edu Abstract This paper describes the open-source Phrase-Based Statistical Machine Transla- tion Decoder - Phramer. The paper also presents the UTD (HLTRI) system build for the WMT06 shared task. Our goal was to improve the translation quality by en- hancing the translation table and by pre- processing the source language text 1 Introduction Despite the fact that the research in Statistical Machine Translation (SMT) is very active, there isn’t an abundance of open-source tools available to the community. In this paper, we present Phramer, an open-source system that embeds a phrase-based decoder, a minimum error rate train- ing (Och, 2003) module and various tools related to Machine Translation (MT). The software is re- leased under BSD license and it is available at http://www.phramer.org/. We also describe our Phramer-based system that we build for the WMT06 shared task. 2 Phramer Phramer is a phrase-based SMT system written in Java. It includes: • A decoder that is compatible with Pharaoh (Koehn, 2004), • A minimum error rate training (MERT) mod- ule, compatible with Phramer’s decoder, with Pharaoh and easily adaptable to other SMT or non-SMT tasks and • various tools. The decoder is fully compatible with Pharaoh 1.2 in the algorithms that are implemented, input files (configuration file, translation table, language models) and command line. Some of the advantages of Phramer over Pharaoh are: (1) source code availability and its permissive license; (2) it is very fast (1.5–3 times faster for most of the configura- tions); (3) it can work with various storage layers for the translation table (TT) and the language models (LMs): memory, remote (access through TCP/IP), disk (using SQLite databases1). Extensions for other storage layers can be very easily implemented; (4) it is more configurable; (5) it accepts compressed data files (TTs and LMs); (6) it is very easy to extend; an example is provided in the package – part-of-speech decoding on either source language, target language or both; support for POS-based language models; (7) it can internally generate n-best lists. Thus no external tools are required. The MERT module is a highly modular, efficient and customizable implementation of the algorithm described in (Och, 2003). The release has imple- mentations for BLEU (Papineni et al., 2002), WER and PER error criteria and it has decoding interfaces for Phramer and Pharaoh. It can be used to search parameters over more than one million vari- ables. It offers features as resume search, reuse hy- potheses from previous runs and various strategies to search for optimal λ weight vectors. 1http://www.sqlite.org/ 146 The package contains a set of tools that include: • Distributed decoding (compatible with both Phramer and Pharaoh) – it automatically splits decoding jobs and distributes them to workers and …

Content truncated to 500 words