Discriminative Training and Maximum Entropy Models for Statistical Machine Translation

Discriminative Training and Maximum Entropy Models for Statistical Machine Translation Franz Josef Och and Hermann Ney Lehrstuhl f¨ur Informatik VI, Computer Science Department RWTH Aachen - University of Technology D-52056 Aachen, Germany {och,ney}@informatik.rwth-aachen.de Abstract We present a framework for statistical machine translation of natural languages based on direct maximum entropy mod- els, which contains the widely used sour- ce-channel approach as a special case. All knowledge sources are treated as feature functions, which depend on the source language sentence, the target language sentence and possible hidden variables. This approach allows a baseline machine translation system to be extended easily by adding new feature functions. We show that a baseline statistical machine transla- tion system is significantly improved us- ing this approach. 1 Introduction We are given a source (‘French’) sentence fJ 1 f1, . . . , fj, . . . , fJ, which is to be translated into a= target (‘English’) sentence eI 1 = e1, . . . , ei, . . . , eI. Among all possible target sentences, we will choose the sentence with the highest probability:1 ˆeI 1 = argmax eI 1 {Pr(eI 1|fJ 1 )} (1) The argmax operation denotes the search problem, i.e. the generation of the output sentence in the target language. 1The notational convention will be as follows. We use the symbol Pr(·) to denote general probability distributions with (nearly) no specific assumptions. In contrast, for model-based probability distributions, we use the generic symbol p(·). 1.1 Source-Channel Model According to Bayes’ decision rule, we can equiva- lently to Eq. 1 perform the following maximization: ˆeI 1 = argmax eI 1 {Pr(eI 1) · Pr(fJ 1 |eI 1)} (2) This approach is referred to as source-channel ap- proach to statistical MT. Sometimes, it is also re- ferred to as the ‘fundamental equation of statisti- cal MT’ (Brown et al., 1993). Here, Pr(eI 1) is the language model of the target language, whereas Pr(fJ 1 |eI 1) is the translation model. Typically, Eq. 2 is favored over the direct translation model of Eq. 1 with the argument that it yields a modular approach. Instead of modeling one probability distribution, we obtain two different knowledge sources that are trained independently. The overall architecture of the source-channel ap- proach is summarized in Figure 1. In general, as shown in this figure, there may be additional trans- formations to make the translation task simpler for the algorithm. Typically, training is performed by applying a maximum likelihood approach. If the language model Pr(eI 1) = pγ(eI 1) depends on pa- rameters γ and the translation model Pr(fJ 1 |eI 1) = pθ(fJ 1 |eI 1) depends on parameters θ, then the opti- mal parameter values are obtained by maximizing the likelihood on a parallel training corpus fS 1 , eS 1 (Brown et al., 1993): ˆθ = argmax θ S � s=1 pθ(fs|es) (3) ˆγ = argmax γ S � s=1 pγ(es) (4) Computational Linguistics (ACL), Philadelphia, July 2002, pp. 295-302. Proceedings of the 40th Annual Meeting …

Content truncated to 500 words