Locally Non-Linear Learning for Statistical Machine Translation via Discretization and Structured Regularization

Locally Non-Linear Learning for Statistical Machine Translation via Discretization and Structured Regularization Jonathan H. Clark∗ Chris Dyer† Alon Lavie† *Microsoft Research †Carnegie Mellon University Redmond, WA 98052, USA Pittsburgh, PA 15213, USA jonathan.clark@microsoft.com {cdyer,alavie}@cs.cmu.edu Abstract Linear models, which support efficient learn- ing and inference, are the workhorses of statis- tical machine translation; however, linear de- cision rules are less attractive from a modeling perspective. In this work, we introduce a tech- nique for learning arbitrary, rule-local, non- linear feature transforms that improve model expressivity, but do not sacrifice the efficient inference and learning associated with linear models. To demonstrate the value of our tech- nique, we discard the customary log transform of lexical probabilities and drop the phrasal translation probability in favor of raw counts. We observe that our algorithm learns a vari- ation of a log transform that leads to better translation quality compared to the explicit log transform. We conclude that non-linear re- sponses play an important role in SMT, an ob- servation that we hope will inform the efforts of feature engineers. 1 Introduction Linear models using log-transformed probabilities as features have emerged as the dominant model in MT systems. This practice can be traced back to the IBM noisy channel models (Brown et al., 1993), which decompose decoding into the product of a translation model (TM) and a language model (LM), motivated by Bayes’ Rule. When Och and Ney (2002) introduced a log-linear model for trans- lation (a linear sum of log-space features), they noted that the noisy channel model was a special case of their model using log probabilities. This ∗This work was conducted as part of the first author’s Ph.D. work at Carnegie Mellon University. same formulation persisted even after the introduc- tion of MERT (Och, 2003), which optimizes a lin- ear model; again, using two log probability fea- tures (TM and LM) with equal weight recovered the noisy channel model. Yet systems now use many more features, some of which are not even probabil- ities. We no longer believe that equal weights be- tween the TM and LM provides optimal translation quality; the probabilities in the TM do not obey the chain rule nor Bayes’ rule, nullifying several the- oretical mathematical justifications for multiplying probabilities. The story of multiplying probabilities may just amount to heavily penalizing small values. The community has abandoned the original mo- tivations for a linear interpolation of two log- transformed features. Is there empirical evidence that we should continue using this particular trans- formation? Do we have any reason to believe it is better than other non-linear transformations? To an- swer these, we explore the issue of non-linearity in models for MT. In the process, we will discuss the impact of linearity on feature engineering and de- velop a general mechanism for learning a class of non-linear transformations of real-valued features. Applying a non-linear transformation such as log to features is one way of achieving a non-linear response function, even if those features are aggre- gated in a linear …

Content truncated to 500 words