Discriminative Training of 150 Million Translation Parameters and Its Application to Pruning

Proceedings of NAACL-HLT 2013, pages 335–341, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Discriminative Training of 150 Million Translation Parameters and Its Application to Pruning Hendra Setiawan and Bowen Zhou IBM T.J. Watson Research Center Yorktown Heights, NY 10598, USA {hendras, zhou}@us.ibm.com Abstract Until recently, the application of discrimina- tive training to log linear-based statistical ma- chine translation has been limited to tuning the weights of a limited number of features or training features with a limited number of pa- rameters. In this paper, we propose to scale up discriminative training of (He and Deng, 2012) to train features with 150 million pa- rameters, which is one order of magnitude higher than previously published effort, and to apply discriminative training to redistribute probability mass that is lost due to model pruning. The experimental results confirm the effectiveness of our proposals on NIST MT06 set over a strong baseline. 1 Introduction State-of-the-art statistical machine translation sys- tems based on a log-linear framework are parame- terized by {λ, Φ}, where the feature weights λ are discriminatively trained (Och and Ney, 2002; Chi- ang et al., 2008b; Simianer et al., 2012) by directly optimizing them against a translation-oriented met- ric such as BLEU. The feature parameters Φ can be roughly divided into two categories: dense fea- ture that measures the plausibility of each translation rule from a particular aspect, e.g., the rule transla- tion probabilities p(f|e) and p(e|f); and sparse fea- ture that fires when certain phenomena is observed, e.g., when a frequent word pair co-occured in a rule. In contrast to λ, feature parameters in Φ are usually modeled by generative models for dense features, or by indicator functions for sparse ones. It is therefore desirable to train the dense features for each rule in a discriminative fashion to maximize some translation criterion. The maximum expected BLEU training of (He and Deng, 2012) is a recent effort towards this direction, and in this paper, we extend their work to a scaled-up task of discriminative training of the features of a strong hierarchical phrase-based model and confirm its effectiveness empirically. In this work, we further consider the application of discriminative training to pruned model. Various pruning techniques (Johnson et al., 2007; Zens et al., 2012; Eck et al., 2007; Lee et al., 2012; Tomeh et al., 2011) have been proposed recently to filter transla- tion rules. One common consequence of pruning is that the probability distribution of many surviving rules become deficient, i.e. � f p(f|e) < 1. In prac- tice, others have chosen either to leave the pruned rules as it-is, or simply to re-normalize the proba- bility mass by distributing the pruned mass to sur- viving rules proportionally. We argue that both ap- proaches are suboptimal, and propose a more prin- cipled method to re-distribute the probability mass, i.e. using discriminative training with some trans- lation criterion. Our experimental results demon- strate that at various pruning levels, our approach improves performance consistently. Particularly at the level of 50% …

Content truncated to 500 words