Tera-Scale Translation Models via Pattern Matching

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 505–512 Manchester, August 2008 Tera-Scale Translation Models via Pattern Matching Adam Lopez School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, United Kingdom alopez@inf.ed.ac.uk Abstract Translation model size is growing at a pace that outstrips improvements in computing power, and this hinders research on many interesting models. We show how an al- gorithmic scaling technique can be used to easily handle very large models. Us- ing this technique, we explore several large model variants and show an improvement 1.4 BLEU on the NIST 2006 Chinese- English task. This opens the door for work on a variety of models that are much less constrained by computational limitations. 1 Introduction Translation model size is growing quickly due to the use of larger training corpora and more com- plex models. As an example of the growth in avail- able training data, consider the curated Europarl corpus (Koehn, 2005), which more than doubled in size from 20 to 44 million words between 2003 and 2007.1 As an example of model complexity, con- sider the popular hierarchical phrase-based model of Chiang (2007), which can translate discontigu- ous phrases. Under the loosest interpretation of this capability, any subset of words in a sentence This research was conducted while I was at the University of Maryland. I thank David Chiang, Bonnie Dorr, Doug Oard, Philip Resnik, and the anonymous reviewers for comments, and especially Chris Dyer for many helpful discussions and for running the Moses experiments. This research was sup- ported by the GALE program of the Defense Advanced Re- search Projects Agency, Contract No. HR0011-06-2-001 and by the EuroMatrix project funded by the European Commis- sion (6th Framework Programme). c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. 1Statistics from http://www.statmt.org/europarl. can be a phrase. Therefore, the number of rules that the model can learn is exponential in sentence length unless strict heuristics are used, which may limit the model’s effectiveness. Many other mod- els translate discontiguous phrases, and the size of their extracted rulesets is such a pervasive problem that it is a recurring topic in the literature (Chiang, 2007; DeNeefe et al., 2007; Simard et al., 2005). Most decoder implementations assume that all model rules and parameters are known in advance. With very large models, computing all rules and parameters can be very slow. This is a bottleneck in experimental settings where we wish to explore many model variants, and therefore presents a real impediment to full exploration of their potential. We present a solution to this problem. To fully motivate the discussion, we give a con- crete example of a very large model, which we generate using simple techniques that are known to improve translation accuracy. The model takes 77 CPU days to compute and consumes nearly a terabyte of external storage (§2). We show how to solve the problem with a previously developed algorithmic scaling technique that we call transla- …

Content truncated to 500 words