Max-Violation Perceptron and Forced Decoding for Scalable MT Training

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1112–1123, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Max-Violation Perceptron and Forced Decoding for Scalable MT Training Heng Yu1∗ 1Institute of Computing Tech. Chinese Academy of Sciences yuheng@ict.ac.cn Liang Huang2† Haitao Mi3 2Queens College & Grad. Center City University of New York {huang@cs.qc,kzhao@gc}.cuny.edu Kai Zhao2 3T.J. Watson Research Center IBM hmi@us.ibm.com Abstract While large-scale discriminative training has triumphed in many NLP problems, its defi- nite success on machine translation has been largely elusive. Most recent efforts along this line are not scalable (training on the small dev set with features from top ∼100 most fre- quent words) and overly complicated. We in- stead present a very simple yet theoretically motivated approach by extending the recent framework of “violation-fixing perceptron”, using forced decoding to compute the target derivations. Extensive phrase-based transla- tion experiments on both Chinese-to-English and Spanish-to-English tasks show substantial gains in BLEU by up to +2.3/+2.0 on dev/test over MERT, thanks to 20M+ sparse features. This is the first successful effort of large-scale online discriminative training for MT. 1 Introduction Large-scale discriminative training has witnessed great success in many NLP problems such as pars- ing (McDonald et al., 2005) and tagging (Collins, 2002), but not yet for machine translation (MT) de- spite numerous recent efforts. Due to scalability is- sues, most of these recent methods can only train on a small dev set of about a thousand sentences rather than on the full training set, and only with 2,000–10,000 rather “dense-like” features (either unlexicalized or only considering highest-frequency words), as in MIRA (Watanabe et al., 2007; Chiang et al., 2008; Chiang, 2012), PRO (Hopkins and May, 2011), and RAMP (Gimpel and Smith, 2012). How- ever, it is well-known that the most important fea- tures for NLP are lexicalized, most of which can not ∗ Work done while visiting City University of New York. †Corresponding author. be seen on a small dataset. Furthermore, these meth- ods often involve complicated loss functions and intricate choices of the “target” derivations to up- date towards or against (e.g. k-best/forest oracles, or hope/fear derivations), and are thus hard to replicate. As a result, the classical method of MERT (Och, 2003) remains the default training algorithm for MT even though it can only tune a handful of dense fea- tures. See also Section 6 for other related work. As a notable exception, Liang et al. (2006) do train a structured perceptron model on the train- ing data with sparse features, but fail to outperform MERT. We argue this is because structured percep- tron, like many structured learning algorithms such as CRF and MIRA, assumes exact search, and search errors inevitably break theoretical properties such as convergence (Huang et al., 2012). Empirically, it is now well accepted that standard perceptron per- forms poorly when search error is severe (Collins and Roark, 2004; Zhang et al., 2013). To address the search error problem we propose a very simple approach based …

Content truncated to 500 words