Bringing Order to Neural Word Embeddings with Embeddings Augmented by Random Permutations (EARP)

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 465–475 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 465 Bringing Order to Neural Word Embeddings with Embeddings Augmented by Random Permutations (EARP) Trevor Cohen Biomedical and Health Informatics University of Washington, Seattle cohenta@uw.edu Dominic Widdows Grab, Inc. Seattle, WA dominic.widdows@grab.com Abstract Word order is clearly a vital part of human language, but it has been used comparatively lightly in distributional vector models. This paper presents a new method for incorporat- ing word order information into word vector embedding models by combining the bene- fits of permutation-based order encoding with the more recent method of skip-gram with negative sampling. The new method intro- duced here is called Embeddings Augmented by Random Permutations (EARP). It operates by applying permutations to the coordinates of context vector representations during the pro- cess of training. Results show an 8% improve- ment in accuracy on the challenging Bigger Analogy Test Set, and smaller but consistent improvements on other analogy reference sets. These findings demonstrate the importance of order-based information in analogical retrieval tasks, and the utility of random permutations as a means to augment neural embeddings. 1 Introduction The recognition of the utility of corpus-derived distributed representations of words for a broad range of Natural Language Processing (NLP) tasks (Collobert et al., 2011) has led to a resur- gence of interest in methods of distributional se- mantics. In particular, the neural-probabilistic word representations produced by the Skip-gram and Continuous Bag-of-Words (CBOW) architec- tures (Mikolov et al., 2013a) implemented in the word2vec and fastText software packages have been extensively evaluated in recent years. As was the case with preceding distributional models (see for example (Sch¨utze, 1993; Lund and Burgess, 1996; Sch¨utze, 1998; Karlgren and Sahlgren, 2001)), these architectures generate vec- tor representations of words — or word embed- dings — such that words with similar proximal neighboring terms within a corpus of text will have similar vector representations. As the relative po- sition of these neighboring terms is generally not considered, distributional models of this nature are often (and sometimes derisively) referred to as bag-of-words models. While methods of encod- ing word order into neural-probabilistic represen- tations have been evaluated, these methods gener- ally require learning additional parameters, either for each position in the sliding window (Mnih and Kavukcuoglu, 2013), or for each context word-by- position pair (Trask et al., 2015; Ling et al., 2015). In this paper we evaluate an alternative method of encoding the position of words within a sliding window into neural word embeddings using Em- beddings Augmented by Random Permutations (EARP). EARP leverages random permutations of context vector representations (Sahlgren et al., 2008), a technique that has not been applied dur- ing the training of neural word embeddings previ- ously. Unlike prior approaches to encoding posi- tion into neural word embeddings, it imposes no computational and negligible space requirements. Results show that the word order information encoded through EARP leads to a nearly 8% …

Content truncated to 500 words