Fast, Easy, and Cheap: Construction of Statistical Machine Translation Models with MapReduce

Proceedings of the Third Workshop on Statistical Machine Translation, pages 199–207, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Fast, Easy, and Cheap: Construction of Statistical Machine Translation Models with MapReduce Christopher Dyer, Aaron Cordova, Alex Mont, Jimmy Lin Laboratory for Computational Linguistics and Information Processing University of Maryland College Park, MD 20742, USA redpony@umd.edu Abstract In recent years, the quantity of parallel train- ing data available for statistical machine trans- lation has increased far more rapidly than the performance of individual computers, re- sulting in a potentially serious impediment to progress. Parallelization of the model- building algorithms that process this data on computer clusters is fraught with challenges such as synchronization, data exchange, and fault tolerance. However, the MapReduce programming paradigm has recently emerged as one solution to these issues: a powerful functional abstraction hides system-level de- tails from the researcher, allowing programs to be transparently distributed across potentially very large clusters of commodity hardware. We describe MapReduce implementations of two algorithms used to estimate the parame- ters for two word alignment models and one phrase-based translation model, all of which rely on maximum likelihood probability esti- mates. On a 20-machine cluster, experimental results show that our solutions exhibit good scaling characteristics compared to a hypo- thetical, optimally-parallelized version of cur- rent state-of-the-art single-core tools. 1 Introduction Like many other NLP problems, output quality of statistical machine translation (SMT) systems in- creases with the amount of training data. Brants et al. (2007) demonstrated that increasing the quantity of training data used for language modeling signifi- cantly improves the translation quality of an Arabic- English MT system, even with far less sophisticated backoff models. However, the steadily increas- ing quantities of training data do not come with- out cost. Figure 1 shows the relationship between the amount of parallel Arabic-English training data used and both the translation quality of a state-of- the-art phrase-based SMT system and the time re- quired to perform the training with the widely-used Moses toolkit on a commodity server.1 Building a model using 5M sentence pairs (the amount of Arabic-English parallel text publicly available from the LDC) takes just over two days.2 This represents an unfortunate state of affairs for the research com- munity: excessively long turnaround on experiments is an impediment to research progress. It is clear that the needs of machine translation re- searchers have outgrown the capabilities of individ- ual computers. The only practical recourse is to dis- tribute the computation across multiple cores, pro- cessors, or machines. The development of parallel algorithms involves a number of tradeoffs. First is that of cost: a decision must be made between “ex- otic” hardware (e.g., large shared memory machines, InfiniBand interconnect) and commodity hardware. There is significant evidence (Barroso et al., 2003) that solutions based on the latter are more cost ef- fective (and for resource-constrained academic in- stitutions, often the only option). Given appropriate hardware, MT researchers must still contend with the challenge of developing software. Quite simply, parallel programming is dif- …

Content truncated to 500 words