Large-Scale Discriminative Training for Statistical Machine Translation Using Held-Out Line Search

Proceedings of NAACL-HLT 2013, pages 248–258, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Large-Scale Discriminative Training for Statistical Machine Translation Using Held-Out Line Search Jeffrey Flanigan Chris Dyer Jaime Carbonell Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA {jflanigan,cdyer,jgc}@cs.cmu.edu Abstract We introduce a new large-scale discrimina- tive learning algorithm for machine translation that is capable of learning parameters in mod- els with extremely sparse features. To ensure their reliable estimation and to prevent over- fitting, we use a two-phase learning algorithm. First, the contribution of individual sparse fea- tures is estimated using large amounts of par- allel data. Second, a small development cor- pus is used to determine the relative contri- butions of the sparse features and standard dense features. Not only does this two-phase learning approach prevent overfitting, the sec- ond pass optimizes corpus-level BLEU of the Viterbi translation of the decoder. We demon- strate significant improvements using sparse rule indicator features in three different trans- lation tasks. To our knowledge, this is the first large-scale discriminative training algo- rithm capable of showing improvements over the MERT baseline with only rule indicator features in addition to the standard MERT fea- tures. 1 Introduction This paper is about large scale discriminative training of machine translation systems. Like MERT (Och, 2003), our procedure directly optimizes the cost of the Viterbi output on corpus-level met- rics, but does so while scaling to millions of features. The training procedure, which we call the Held-Out Line Search algorithm (HOLS), is a two-phase iter- ative batch optimization procedure consisting of (1) a gradient calculation on a differentiable approxima- tion to the loss on a large amount of parallel training data and (2) a line search (using the standard MERT algorithm) to search in a subspace defined by the gradient for the weights that minimize the true cost. While sparse features are successfully used in many NLP systems, such parameterizations pose a number of learning challenges. First, since any one feature is likely to occur infrequently, a large amount of training data is necessary to reliably estimate their weights. Therefore, we use the full parallel train- ing data (rather than a small development set) to estimate the contribution of the sparse features in phase 1. Second, sparse features can lead to overfit- ting. To prevent this from hurting our model’s ability to generalize to new data, we do two things. First, we use “grammar and language model folds” (trans- lation grammars and language models built from other portions of the training data than are being used for discriminative training), and second, we run the phase 2 line search on a held-out develop- ment set. Finally, since our algorithm requires de- coding the entire training corpus, it is desirable (on computational grounds) to only require one or two passes through the training data. To get the most out of these passes, we rescale features by their inverse frequency which improves the scaling of the opti- mization problem. In addition to learning with …

Content truncated to 500 words