Phrasal: A Toolkit for New Directions in Statistical Machine Translation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 114–121, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Phrasal: A Toolkit for New Directions in Statistical Machine Translation Spence Green, Daniel Cer, and Christopher D. Manning Computer Science Department, Stanford University {spenceg,danielcer,manning}@stanford.edu Abstract We present a new version of Phrasal, an open-source toolkit for statistical phrase- based machine translation. This revision includes features that support emerging re- search trends such as (a) tuning with large feature sets, (b) tuning on large datasets like the bitext, and (c) web-based interactive ma- chine translation. A direct comparison with Moses shows favorable results in terms of decoding speed and tuning time. 1 Introduction In the early part of the last decade, phrase-based ma- chine translation (MT) (Koehn et al., 2003) emerged as the preeminent design of statistical MT systems. However, most systems were proprietary or closed- source, so progress was initially constrained by the high engineering barrier to entry into the field. Then Moses (Koehn et al., 2007) was released. What followed was a flowering of work on all as- pects of the translation problem, from rule extrac- tion to deployment issues. Other toolkits appeared including Joshua (Post et al., 2013), Jane (Wuebker et al., 2012), cdec (Dyer et al., 2010) and the first version of our package, Phrasal (Cer et al., 2010), a Java-based, open source package. This paper presents a completely re-designed release of Phrasal that lowers the barrier to entry into several exciting areas of MT research. First, Phrasal exposes a simple yet flexible feature API for building large-scale, feature-rich systems. Second, Phrasal provides multi-threaded decoding and on- line tuning for learning feature-rich models on very large datasets, including the bitext. Third, Phrasal supplies the key ingredients for web-based, inter- active MT: an asynchronous RESTful JSON web service implemented as a J2EE servlet, integrated pre- and post-processing, and fast search. Revisions to Phrasal were guided by several de- sign choices. First, we optimized the system for multi-core architectures, eschewing distributed in- frastructure like Hadoop and MapReduce. While “scaling-out” with distributed infrastructure is the conventional industry and academic choice, we find that “scaling-up” on a single large-node is an at- tractive yet overlooked alternative (Appuswamy et al., 2013). A single “scale-up” node is usually competitive in terms of cost and performance, and multi-core code has fewer dependencies in terms of software and expertise. Second, Phrasal makes extensive use of Java interfaces and reflection. This is especially helpful in the feature API. A feature function can be added to the system by simply im- plementing an interface and specifying the class name on the decoder command line. There is no need to modify or recompile anything other than the new feature function. This paper presents a direct comparison of Phrasal and Moses that shows favorable results in terms of decoding speed and tuning time. An indirect comparison via the WMT2014 shared task (Neidert et al., 2014) showed that Phrasal compares favorably to Moses in an evaluation setting. The …

Content truncated to 500 words