Large Language Models in Machine Translation

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 858–867, Prague, June 2007. c⃝2007 Association for Computational Linguistics Large Language Models in Machine Translation Thorsten Brants Ashok C. Popat Peng Xu Franz J. Och Jeffrey Dean Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 94303, USA {brants,popat,xp,och,jeff}@google.com Abstract This paper reports on the benefits of large- scale statistical language modeling in ma- chine translation. A distributed infrastruc- ture is proposed which we use to train on up to 2 trillion tokens, resulting in language models having up to 300 billion n-grams. It is capable of providing smoothed probabil- ities for fast, single-pass decoding. We in- troduce a new smoothing method, dubbed Stupid Backoff, that is inexpensive to train on large data sets and approaches the quality of Kneser-Ney Smoothing as the amount of training data increases. 1 Introduction Given a source-language (e.g., French) sentence f, the problem of machine translation is to automati- cally produce a target-language (e.g., English) trans- lation ˆe. The mathematics of the problem were for- malized by (Brown et al., 1993), and re-formulated by (Och and Ney, 2004) in terms of the optimization ˆe = arg max e M � m=1 λmhm(e, f) (1) where {hm(e, f)} is a set of M feature functions and {λm} a set of weights. One or more feature func- tions may be of the form h(e, f) = h(e), in which case it is referred to as a language model. We focus on n-gram language models, which are trained on unlabeled monolingual text. As a general rule, more data tends to yield better language mod- els. Questions that arise in this context include: (1) How might one build a language model that allows scaling to very large amounts of training data? (2) How much does translation performance improve as the size of the language model increases? (3) Is there a point of diminishing returns in performance as a function of language model size? This paper proposes one possible answer to the first question, explores the second by providing learning curves in the context of a particular statis- tical machine translation system, and hints that the third may yet be some time in answering. In particu- lar, it proposes a distributed language model training and deployment infrastructure, which allows direct and efficient integration into the hypothesis-search algorithm rather than a follow-on re-scoring phase. While it is generally recognized that two-pass de- coding can be very effective in practice, single-pass decoding remains conceptually attractive because it eliminates a source of potential information loss. 2 N-gram Language Models Traditionally, statistical language models have been designed to assign probabilities to strings of words (or tokens, which may include punctuation, etc.). Let wL 1 = (w1, . . . , wL) denote a string of L tokens over a fixed vocabulary. An n-gram language model assigns a probability to wL 1 according to P(wL 1 ) = L � i=1 P(wi|wi−1 1 ) ≈ L …

Content truncated to 500 words