Feature Noising for Log-Linear Structured Prediction

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1170–1179, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Feature Noising for Log-linear Structured Prediction Sida I. Wang∗, Mengqiu Wang∗, Stefan Wager†, Percy Liang, Christopher D. Manning Department of Computer Science, †Department of Statistics Stanford University, Stanford, CA 94305, USA {sidaw, mengqiu, pliang, manning}@cs.stanford.edu swager@stanford.edu Abstract NLP models have many and sparse features, and regularization is key for balancing model overfitting versus underfitting. A recently re- popularized form of regularization is to gen- erate fake training data by repeatedly adding noise to real data. We reinterpret this noising as an explicit regularizer, and approximate it with a second-order formula that can be used during training without actually generating fake data. We show how to apply this method to structured prediction using multinomial lo- gistic regression and linear-chain CRFs. We tackle the key challenge of developing a dy- namic program to compute the gradient of the regularizer efficiently. The regularizer is a sum over inputs, so we can estimate it more accurately via a semi-supervised or transduc- tive extension. Applied to text classification and NER, our method provides a >1% abso- lute performance gain over use of standard L2 regularization. 1 Introduction NLP models often have millions of mainly sparsely attested features. As a result, balancing overfitting versus underfitting through good weight regulariza- tion remains a key issue for achieving optimal per- formance. Traditionally, L2 or L1 regularization is employed, but these simple types of regularization penalize all features in a uniform way without tak- ing into account the properties of the actual model. An alternative approach to regularization is to generate fake training data by adding random noise to the input features of the original training data. In- tuitively, this can be thought of as simulating miss- ∗Both authors contributed equally to the paper ing features, whether due to typos or use of a pre- viously unseen synonym. The effectiveness of this technique is well-known in machine learning (Abu- Mostafa, 1990; Burges and Sch¨olkopf, 1997; Simard et al., 2000; Rifai et al., 2011a; van der Maaten et al., 2013), but working directly with many cor- rupted copies of a dataset can be computationally prohibitive. Fortunately, feature noising ideas often lead to tractable deterministic objectives that can be optimized directly. Sometimes, training with cor- rupted features reduces to a special form of reg- ularization (Matsuoka, 1992; Bishop, 1995; Rifai et al., 2011b; Wager et al., 2013). For example, Bishop (1995) showed that training with features that have been corrupted with additive Gaussian noise is equivalent to a form of L2 regularization in the low noise limit. In other cases it is possible to develop a new objective function by marginalizing over the artificial noise (Wang and Manning, 2013; van der Maaten et al., 2013). The central contribution of this paper is to show how to efficiently simulate training with artificially noised features in the context of log-linear struc- tured prediction, without actually having to gener- …

Content truncated to 500 words