An Empirical Comparison of Features and Tuning for Phrase-based Machine Translation

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 466–476, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics An Empirical Comparison of Features and Tuning for Phrase-based Machine Translation Spence Green, Daniel Cer, and Christopher D. Manning Computer Science Department, Stanford University {spenceg,danielcer,manning}@stanford.edu Abstract Scalable discriminative training methods are now broadly available for estimating phrase-based, feature-rich translation mod- els. However, the sparse feature sets typi- cally appearing in research evaluations are less attractive than standard dense features such as language and translation model probabilities: they often overfit, do not gen- eralize, or require complex and slow fea- ture extractors. This paper introduces ex- tended features, which are more specific than dense features yet more general than lexicalized sparse features. Large-scale ex- periments show that extended features yield robust BLEU gains for both Arabic-English (+1.05) and Chinese-English (+0.67) rel- ative to a strong feature-rich baseline. We also specialize the feature set to specific data domains, identify an objective function that is less prone to overfitting, and release fast, scalable, and language-independent tools for implementing the features. 1 Introduction Scalable discriminative algorithm design for ma- chine translation (MT) has lately been a booming enterprise. There are now algorithms for every taste: probabilistic and distribution-free, online and batch, regularized and unregularized. Technical differ- ences aside, the papers that apply these algorithms to phrase-based translation often share a curious empirical characteristic: the algorithms support ex- tra features, but the features do not significantly improve translation. For example, Hopkins and May (2011) showed that PRO with some simple ad hoc features only exceeds the baseline on one of three language pairs. Gimpel and Smith (2012b) observed a similar result for both PRO and their ramp-loss algorithm. Cherry and Foster (2012) found that, at least in the batch case, many algo- rithms produce similar results, and features only significantly increased quality for one of three lan- guage pairs. Only recently did Cherry (2013) and Green et al. (2013b) identify certain features that consistently reduce error. These empirical results suggest that feature de- sign and model fitting, the subjects of this paper, warrant a closer look. We introduce an effective extended feature set for phrase-based MT and iden- tify a loss function that is less prone to overfitting. Extended features share three attractive characteris- tics with the standard Moses dense features (Koehn et al., 2007): ease of implementation, language in- dependence, and independence from ancillary cor- pora like treebanks. In our experiments, they do not overfit and can be extracted efficiently during decoding. Because all feature weights are tuned on the development set, the new feature templates are amenable to feature augmentation (Daumé III, 2007), a simple domain adaptation technique that we show works surprisingly well for MT. Extended features are designed according to a principle rather than a rule: they should fire less than standard dense features, which are general, but more than so-called sparse features, which are very specific—they are usually lexicalized—and thus prone to overfitting. This principle is motivated by …

Content truncated to 500 words