Robust Gram Embeddings

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1060–1065, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Robust Gram Embeddings Taygun Kekec¸ and D.M.J. Tax Pattern Recognition and Bioinformatics Laboratory Delft University of Technology Delft, 2628CD, The Netherlands taygunkekec@gmail.com,D.M.J.Tax@tudelft.nl Abstract Word embedding models learn vectorial word representations that can be used in a variety of NLP applications. When training data is scarce, these models risk losing their gener- alization abilities due to the complexity of the models and the overfitting to finite data. We propose a regularized embedding formu- lation, called Robust Gram (RG), which pe- nalizes overfitting by suppressing the dispar- ity between target and context embeddings. Our experimental analysis shows that the RG model trained on small datasets generalizes better compared to alternatives, is more robust to variations in the training set, and correlates well to human similarities in a set of word sim- ilarity tasks. 1 Introduction Word embeddings represent each word as a unique vector in a linear vector space, encoding particular semantic and syntactic structure of the natural lan- guage (Arora et al., 2016). In various lingual tasks, these sequence prediction models shown superior re- sults over the traditional count-based models (Ba- roni et al., 2014). Tasks such as sentiment analysis (Maas et al., 2011) and sarcasm detection (Ghosh et al., 2015) enjoys the merits of these features. These word embeddings optimize features and predictors simultaneously, which can be interpreted as a factorization of the word cooccurence matrix C. In most realistic scenarios these models have to be learned from a small training set. Furthermore, word distributions are often skewed, and optimiz- ing the reconstruction of ˆC puts too much empha- sis on the high frequency pairs (Levy and Goldberg, 2014). On the other hand, by having an unlucky and scarce data sample, the estimated ˆC rapidly deviates from the underlying true cooccurence, in particu- lar for low-frequency pairs (Lemaire and Denhire, 2008). Finally, noise (caused by stemming, removal of high frequency pairs, typographical errors, etc.) can increase the estimation error heavily (Arora et al., 2015). It is challenging to derive a computationally tractable algorithm that solves all these problems. Spectral factorization approaches usually employ Laplace smoothing or a type of SVD weighting to alleviate the effect of the noise (Turney and Pantel, 2010). Alternatively, iteratively optimized embed- dings such as Skip Gram (SG) model (Mikolov et al., 2013b) developed various mechanisms such as undersampling of highly frequent hub words apriori, and throwing rare words out of the training. Here we propose a fast, effective and general- izable embedding approach, called Robust Gram, that penalizes complexity arising from the factorized embedding spaces. This design alleviates the need from tuning the aforementioned pseudo-priors and the preprocessing procedures. Experimental results show that our regularized model 1) generalizes bet- ter given a small set of samples while other methods yield insufficient generalization 2) is more robust to arbitrary perturbations in the sample set and alterna- tions in the preprocessing specifications 3) …

Content truncated to 500 words