Online Relative Margin Maximization for Statistical Machine Translation

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1116–1126, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Online Relative Margin Maximization for Statistical Machine Translation Vladimir Eidelman Computer Science and UMIACS University of Maryland College Park, MD vlad@umiacs.umd.edu Yuval Marton Microsoft City Center Plaza Bellevue, WA yuvalmarton@gmail.com Philip Resnik Linguistics and UMIACS University of Maryland College Park, MD resnik@umd.edu Abstract Recent advances in large-margin learning have shown that better generalization can be achieved by incorporating higher order information into the optimization, such as the spread of the data. However, these so- lutions are impractical in complex struc- tured prediction problems such as statis- tical machine translation. We present an online gradient-based algorithm for rela- tive margin maximization, which bounds the spread of the projected data while max- imizing the margin. We evaluate our op- timizer on Chinese-English and Arabic- English translation tasks, each with small and large feature sets, and show that our learner is able to achieve significant im- provements of 1.2-2 BLEU and 1.7-4.3 TER on average over state-of-the-art opti- mizers with the large feature set. 1 Introduction The desire to incorporate high-dimensional sparse feature representations into statistical machine translation (SMT) models has driven recent re- search away from Minimum Error Rate Training (MERT) (Och, 2003), and toward other discrim- inative methods that can optimize more features. Examples include minimum risk (Smith and Eis- ner, 2006), pairwise ranking (PRO) (Hopkins and May, 2011), RAMPION (Gimpel and Smith, 2012), and variations of the margin-infused relaxation al- gorithm (MIRA) (Watanabe et al., 2007; Chiang et al., 2008; Cherry and Foster, 2012). While the ob- jective function and optimization method vary for each optimizer, they can all be broadly described as learning a linear model, or parameter vector w, which is used to score alternative translation hy- potheses. In every SMT system, and in machine learn- ing in general, the goal of learning is to find a model that generalizes well, i.e. one that will yield good translations for previously unseen sentences. However, as the dimension of the feature space in- creases, generalization becomes increasingly diffi- cult. Since only a small portion of all (sparse) fea- tures may be observed in a relatively small fixed set of instances during tuning, we are prone to overfit the training data. An alternative approach for solving this problem is estimating discrimina- tive feature weights directly on the training bi- text (Tillmann and Zhang, 2006; Blunsom et al., 2008; Simianer et al., 2012), which is usually sub- stantially larger than the tuning set, but this is com- plementary to our goal here of better generaliza- tion given a fixed size tuning set. In order to achieve that goal, we need to care- fully choose what objective to optimize, and how to perform parameter estimation of w for this ob- jective. We focus on large-margin methods such as SVM (Joachims, 1998) and passive-aggressive algorithms such as MIRA. Intuitively these seek a w such that the separating distance in geomet- …

Content truncated to 500 words