Fast and Adaptive Online Training of Feature-Rich Translation Models

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 311–321, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Fast and Adaptive Online Training of Feature-Rich Translation Models Spence Green, Sida Wang, Daniel Cer, and Christopher D. Manning Computer Science Department, Stanford University {spenceg,sidaw,danielcer,manning}@stanford.edu Abstract We present a fast and scalable online method for tuning statistical machine trans- lation models with large feature sets. The standard tuning algorithm—MERT—only scales to tens of features. Recent discrimi- native algorithms that accommodate sparse features have produced smaller than ex- pected translation quality gains in large systems. Our method, which is based on stochastic gradient descent with an adaptive learning rate, scales to millions of features and tuning sets with tens of thousands of sentences, while still converging after only a few epochs. Large-scale experiments on Arabic-English and Chinese-English show that our method produces significant trans- lation quality gains by exploiting sparse fea- tures. Equally important is our analysis, which suggests techniques for mitigating overfitting and domain mismatch, and ap- plies to other recent discriminative methods for machine translation. 1 Introduction Sparse, overlapping features such as words and n- gram contexts improve many NLP systems such as parsers and taggers. Adaptation of discriminative learning methods for these types of features to sta- tistical machine translation (MT) systems, which have historically used idiosyncratic learning tech- niques for a few dense features, has been an active research area for the past half-decade. However, de- spite some research successes, feature-rich models are rarely used in annual MT evaluations. For exam- ple, among all submissions to the WMT and IWSLT 2012 shared tasks, just one participant tuned more than 30 features (Hasler et al., 2012a). Slow uptake of these methods may be due to implementation complexities, or to practical difficulties of configur- ing them for specific translation tasks (Gimpel and Smith, 2012; Simianer et al., 2012, inter alia). We introduce a new method for training feature- rich MT systems that is effective yet comparatively easy to implement. The algorithm scales to millions of features and large tuning sets. It optimizes a lo- gistic objective identical to that of PRO (Hopkins and May, 2011) with stochastic gradient descent, al- though other objectives are possible. The learning rate is set adaptively using AdaGrad (Duchi et al., 2011), which is particularly effective for the mixture of dense and sparse features present in MT models. Finally, feature selection is implemented as efficient L1 regularization in the forward-backward splitting (FOBOS) framework (Duchi and Singer, 2009). Ex- periments show that our algorithm converges faster than batch alternatives. To learn good weights for the sparse features, most algorithms—including ours—benefit from more tuning data, and the natural source is the train- ing bitext. However, the bitext presents two prob- lems. First, it has a single reference, sometimes of lower quality than the multiple references in tun- ing sets from MT competitions. Second, large bi- texts often comprise many text genres (Haddow and Koehn, 2012), a virtue for classical dense MT mod- …

Content truncated to 500 words