Training Non-Parametric Features for Statistical Machine Translation

Proceedings of the Second Workshop on Statistical Machine Translation, pages 72–79, Prague, June 2007. c⃝2007 Association for Computational Linguistics Training Non-Parametric Features for Statistical Machine Translation Patrick Nguyen, Milind Mahajan and Xiaodong He Microsoft Corporation 1 Microsoft Way, Redmond, WA 98052 {panguyen,milindm,xiaohe}@microsoft.com Abstract Modern statistical machine translation sys- tems may be seen as using two components: feature extraction, that summarizes informa- tion about the translation, and a log-linear framework to combine features. In this pa- per, we propose to relax the linearity con- straints on the combination, and hence relax- ing constraints of monotonicity and indepen- dence of feature functions. We expand fea- tures into a non-parametric, non-linear, and high-dimensional space. We extend empir- ical Bayes reward training of model param- eters to meta parameters of feature genera- tion. In effect, this allows us to trade away some human expert feature design for data. Preliminary results on a standard task show an encouraging improvement. 1 Introduction In recent years, statistical machine translation have experienced a quantum leap in quality thanks to au- tomatic evaluation (Papineni et al., 2002) and error- based optimization (Och, 2003). The conditional log-linear feature combination framework (Berger, Della Pietra and Della Pietra, 1996) is remarkably simple and effective in practice. Therefore, re- cent efforts (Och et al., 2004) have concentrated on feature design – wherein more intelligent features may be added. Because of their simplicity, how- ever, log-linear models impose some constraints on how new information may be inserted into the sys- tem to achieve the best results. In other words, new information needs to be parameterized care- fully into one or more real valued feature functions. Therefore, that requires some human knowledge and understanding. When not readily available, this is typically replaced with painstaking experimenta- tion. We propose to replace that step with automatic training of non-parametric agnostic features instead, hopefully relieving the burden of finding the optimal parameterization. First, we define the model and the objective func- tion training framework, then we describe our new non-parametric features. 2 Model In this section, we describe the general log-linear model used for statistical machine translation, as well as a training objective function and algorithm. The goal is to translate a French (source) sentence indexed by t, with surface string ft. Among a set of Kt outcomes, we denote an English (target) a hy- pothesis with surface string e(t) k indexed by k. 2.1 Log-linear Model The prevalent translation model in modern systems is a conditional log-linear model (Och and Ney, 2002). From a hypothesis e(t) k , we extract features h(t) k , abbreviated hk, as a function of e(t) k and ft. The conditional probability of a hypothesis e(t) k given a source sentence ft is: pk ≜ p(e(t) k |ft) ≜ exp[λ · hk] Zft;λ , 72 where the partition function Zft;λ is given by: Zft;λ = � j exp[λ · hj]. The vector of parameters of the model λ, gives a relative importance to each feature function compo- nent. 2.2 Training Criteria In …

Content truncated to 500 words