Single Training Dimension Selection for Word Embedding with PCA

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3597–3602, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3597 Single Training Dimension Selection for Word Embedding with PCA Yu Wang Apple w.y@apple.com Abstract In this paper, we present a fast and reliable method based on PCA to select the number of dimensions for word embeddings. First, we train one embedding with a generous upper bound (e.g. 1,000) of dimensions. Then we transform the embeddings using PCA and in- crementally remove the lesser dimensions one at a time while recording the embeddings’ per- formance on language tasks. Lastly, we se- lect the number of dimensions while balancing model size and accuracy. Experiments using various datasets and language tasks demon- strate that we are able to train 10 times fewer sets of embeddings while retaining optimal performance. Researchers interested in train- ing the best-performing embeddings for down- stream tasks, such as sentiment analysis, ques- tion answering and hypernym extraction, as well as those interested in embedding com- pression should find the method helpful. 1 Introduction Word embeddings constitute an integral part of the implementation of numerous NLP tasks ranging from sentiment classification (Lin et al., 2015), nationality classification (Ye et al., 2017), classi- fication of behavior on Twitter (Wang et al., 2017), to measuring document similarity (Kusner et al., 2015). The strength of word embeddings stems from their embedding words into low dimensional continuous vector space (Mnih and Kavukcuoglu, 2013; Pennington et al., 2014; Tang et al., 2015; Raunak, 2017). Various algorithms have been proposed for learning words’ vector space representations, in- cluding most notably Mikolov et al. (2013b), Pen- nington et al. (2014), and Nickel and Kiela (2017). However, the exact definition of ‘low’ dimension- ality is rarely explored. As pointed out by Yin and Shen (2018), the most frequently used dimension- ality is 300, largely due to the fact that the early influential papers (Mikolov et al., 2013b; Pen- nington et al., 2014) used 300. Other often used dimensionalities include 200 (Tang et al., 2015; Ling et al., 2016; Nickel and Kiela, 2017) and 500 (Tang and Liu, 2009; Perozzi et al., 2014). The impact of vector dimension on embed- dings’ performance is well known (Lai et al., 2016). With too few dimensions, the model will underfit; with too many dimensions the model will overfit. Both undercut the embeddings’ perfor- mance (Yin and Shen, 2018). What is also known is that the size of the embeddings will grow lin- early with the vector dimension (Ling et al., 2016; Raunak, 2017). What is less known is how to identify the optimal vector dimension given any dataset. The method we propose here helps fill this gap. 100 200 300 400 500 # Dimension 30 32 34 36 38 40 Accuracy Word Analogy 0 100 200 300 400 500 # Dimension 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 # Parameters 1e7 Embedding Size Figure 1: …

Content truncated to 500 words