Word Embeddings through Hellinger PCA

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 482–490, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Word Embeddings through Hellinger PCA R´emi Lebret Idiap Research Institute Rue Marconi 19, CP 592 1920 Martigny, Switzerland remi@lebret.ch Ronan Collobert Idiap Research Institute Rue Marconi 19, CP 592 1920 Martigny, Switzerland ronan@collobert.com Abstract Word embeddings resulting from neural language models have been shown to be a great asset for a large variety of NLP tasks. However, such architecture might be difficult and time-consuming to train. Instead, we propose to drastically sim- plify the word embeddings computation through a Hellinger PCA of the word co- occurence matrix. We compare those new word embeddings with some well-known embeddings on named entity recognition and movie review tasks and show that we can reach similar or even better perfor- mance. Although deep learning is not re- ally necessary for generating good word embeddings, we show that it can provide an easy way to adapt embeddings to spe- cific tasks. 1 Introduction Building word embeddings has always generated much interest for linguists. Popular approaches such as Brown clustering algorithm (Brown et al., 1992) have been used with success in a wide vari- ety of NLP tasks (Sch¨utze, 1995; Koo et al., 2008; Ratinov and Roth, 2009). Those word embed- dings are often seen as a low dimensional-vector space where the dimensions are features poten- tially describing syntactic or semantic properties. Recently, distributed approaches based on neural network language models (NNLM) have revived the field of learning word embeddings (Collobert and Weston, 2008; Huang and Yates, 2009; Turian et al., 2010; Collobert et al., 2011). However, a neural network architecture can be hard to train. Finding the right parameters to tune the model is often a challenging task and the training phase is in general computationally expensive. This paper aims to show that such good word embeddings can be obtained using simple (mostly linear) operations. We show that similar word embeddings can be computed using the word co- occurrence statistics and a well-known dimension- ality reduction operation such as Principal Com- ponent Analysis (PCA). We then compare our em- beddings with the CW (Collobert and Weston, 2008), Turian (Turian et al., 2010), HLBL (Mnih and Hinton, 2008) embeddings, which come from deep architectures and the LR-MVL (Dhillon et al., 2011) embeddings, which also come from a spectral method on several NLP tasks. We claim that, assuming an appropriate met- ric, a simple spectral method as PCA can generate word embeddings as good as with deep-learning architectures. On the other hand, deep-learning architectures have shown their potential in sev- eral supervised NLP tasks, by using these word embeddings. As they are usually generated over large corpora of unlabeled data, words are repre- sented in a generic manner. Having generic em- beddings, good performance can be achieved on NLP tasks where the syntactic aspect is domi- nant such as Part-Of-Speech, chunking and NER (Turian et al., 2010; Collobert et al., …

Content truncated to 500 words