Dramatically Reducing Training Data Size Through Vocabulary Saturation

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 281–291, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Dramatically Reducing Training Data Size Through Vocabulary Saturation William D. Lewis Microsoft Research One Microsoft Way Redmond, WA 98052 wilewis@microsoft.com Sauleh Eetemadi Microsoft Research One Microsoft Way, Redmond, WA 98052 Michigan State University, East Lansing, MI 48824 saulehe@microsoft.com Abstract Our field has seen significant improve- ments in the quality of machine translation systems over the past several years. The single biggest factor in this improvement has been the accumulation of ever larger stores of data. However, we now find our- selves the victims of our own success, in that it has become increasingly difficult to train on such large sets of data, due to limitations in memory, processing power, and ultimately, speed (i.e., data to mod- els takes an inordinate amount of time). Some teams have dealt with this by focus- ing on data cleaning to arrive at smaller data sets (Denkowski et al., 2012a; Rarrick et al., 2011), “domain adaptation” to ar- rive at data more suited to the task at hand (Moore and Lewis, 2010; Axelrod et al., 2011), or by specifically focusing on data reduction by keeping only as much data as is needed for building models e.g., (Eck et al., 2005). This paper focuses on tech- niques related to the latter efforts. We have developed a very simple n-gram counting method that reduces the size of data sets dramatically, as much as 90%, and is ap- plicable independent of specific dev and test data. At the same time it reduces model sizes, improves training times, and, because it attempts to preserve contexts for all n-grams in a corpus, the cost in quality is minimal (as measured by BLEU ). Fur- ther, unlike other methods created specif- ically for data reduction that have similar effects on the data, our method scales to very large data, up to tens to hundreds of millions of parallel sentences. 1 Introduction The push to build higher and higher quality Sta- tistical Machine Translation systems has led the efforts to collect more and more data. The English-French (nearly) Gigaword Parallel Corpus (Callison-Burch et al., 2009), which we will refer to henceforth as EnFrGW, is the result of one such effort. The EnFrGW is a publicly available cor- pus scraped from Canadian, European and inter- national Web sites, consisting of over 22.5M par- allel English-French sentences. This corpus has been used regularly in the WMT competition since 2009. As the size of data increases, BLEU scores in- crease, but the increase in BLEU is not linear in re- lation to data size. The relationship between data size and BLEU flattens fairly quickly, as demon- strated in Figure 1. Here we see that BLEU scores increase rapidly with small amounts of data, but they taper off and flatten at much larger amounts. Clearly, as we add more data, the value of the new data diminishes with each increase, until very little value is …

Content truncated to 500 words