Encoding Prior Knowledge with Eigenword Embeddings

Encoding Prior Knowledge with Eigenword Embeddings Dominique Osborne Department of Mathematics and Statistics University of Strathclyde Glasgow, G1 1XH, UK dominique.osborne.13@uni.strath.ac.uk Shashi Narayan and Shay B. Cohen School of Informatics University of Edinburgh Edinburgh, EH8 9LE, UK {snaraya2,scohen}@inf.ed.ac.uk Abstract Canonical correlation analysis (CCA) is a method for reducing the dimension of data represented using two views. It has been previously used to derive word embeddings, where one view indicates a word, and the other view indicates its context. We describe a way to incorporate prior knowledge into CCA, give a theoretical justification for it, and test it by deriving word embeddings and evaluating them on a myriad of datasets. 1 Introduction In recent years there has been an immense in- terest in representing words as low-dimensional continuous real-vectors, namely word embeddings. Word embeddings aim to capture lexico-semantic information such that regularities in the vocabulary are topologically represented in a Euclidean space. Such word embeddings have achieved state-of-the- art performance on many natural language process- ing (NLP) tasks, e.g., syntactic parsing (Socher et al., 2013), word or phrase similarity (Mikolov et al., 2013b), dependency parsing (Bansal et al., 2014), unsupervised learning (Parikh et al., 2014) and oth- ers. Since the discovery that word embeddings are useful as features for various NLP tasks, research on word embeddings has taken on a life of its own, with a vibrant community searching for better word rep- resentations in a variety of problems and datasets. These word embeddings are often induced from large raw text capturing distributional co-occurrence information via neural networks (Bengio et al., 2003; Mikolov et al., 2013b; Mikolov et al., 2013c) or spectral methods (Deerwester et al., 1990; Dhillon et al., 2015). While these general pur- pose word embeddings have achieved significant im- provement in various tasks in NLP, it has been dis- covered that further tuning of these continuous word representations for specific tasks improves their per- formance by a larger margin. For example, in de- pendency parsing, word embeddings could be tai- lored to capture similarity in terms of context within syntactic parses (Bansal et al., 2014) or they could be refined using semantic lexicons such as WordNet (Miller, 1995), FrameNet (Baker et al., 1998) and the Paraphrase Database (Ganitkevitch et al., 2013) to improve various similarity tasks (Yu and Dredze, 2014; Faruqui et al., 2015; Rothe and Sch¨utze, 2015). This paper proposes a method to encode prior semantic knowledge in spectral word embeddings (Dhillon et al., 2015). Spectral learning algorithms are of great inter- est for their speed, scalability, theoretical guaran- tees and performance in various NLP applications. These algorithms are no strangers to word embed- dings either. In latent semantic analysis (LSA, (Deerwester et al., 1990; Landauer et al., 1998)), word embeddings are learned by performing SVD on the word by document matrix. Recently, Dhillon et al. (2015) have proposed to use canonical cor- relation analysis (CCA) as a method to learn low- dimensional real vectors, called Eigenwords. Un- like LSA based methods, CCA based methods are scale …

Content truncated to 500 words