Task-dependent Optimal Weight Combinations for Static Embeddings

Task-dependent Optimal Weight Combinations for Static Embed- dings Nathaniel R. Robinson, Carnegie Mellon University, USA nrrobins@cs.cmu.edu Nathaniel Carlson, Brigham Young University, USA natec18@byu.edu David R. Mortensen, Carnegie Mellon University, USA dmortens@cs.cmu.edu Elizabeth Ann Vargas, Brigham Young University, USA elizag17@byu.edu Thomas Fackrell, Brigham Young University, USA tfac1997@byu.edu Nancy Fulda, Brigham Young University, USA nfulda@cs.byu.edu Abstract A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate up to 12.5% improvements. Notably, GloVe’s default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%. 1 Introduction Static word embeddings are used in a broad range of NLP applications, including conversational gameplay (Andrus and Fulda, 2020), text categorization (Minaee et al., 2021; Mitra et al., 2016), translation (Sabet et al., 2020; Jansen, 2017; Pourdamghani et al., 2018), affor- dance detection (Fulda et al., 2017a), and semantic anal- ysis (Hamilton et al., 2016). In addition to using static embeddings directly, researchers often combine them with contextualized models or use them for embedding initialization of downstream tasks (Kocmi and Bojar, 2017) such as summarization (Lin et al., 2021) and neu- ral machine translation (Qi et al., 2018). The persis- tence of static embeddings is due in part to their ease of use and low computational requirements. Rather than needing a forward pass through a neural network to em- bed each word, pre-trained embeddings can be stored in memory and retrieved with complexity 𝐴���(1).1 §4.2 outlines more advantages of static embeddings. 1This is assuming a default Python or Java hash map. The worst case would be 𝐴��� (𝐴���) with vocabulary size 𝐴���, in the case of a trivially slow hash map, which is still well under transformer-based embed- ding retrieval complexity. We propose an augmentation of three popular em- bedding methods (word2vec skip-gram, GloVe, and fast- Text). Word2vec skip-gram (Mikolov et al., 2013a) is a neural word context predictor, GloVe (Pennington et al., 2014) is a log-bilinear model that includes global con- text information with a co-occurrence matrix, and fast- Text (Bojanowski et al., 2017) incorporates sub-word in- formation via character n-grams with a skip-gram ob- jective to expedite training and handle unseen words. More details about these algorithms are in §2. Each of them produces two separate embedding sets ("tar- get" and "context", see Figure 1) that we combine in previously unexplored ways. We show that these typ- ically unexplored target and …

Content truncated to 500 words