Random Walks and Neural Network Language Models on Knowledge Bases

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1434–1439, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Random Walks and Neural Network Language Models on Knowledge Bases Josu Goikoetxea, Aitor Soroa and Eneko Agirre IXA NLP Group University of the Basque Country Donostia, Basque Country {josu.goikoetxea,a.soroa,e.agirre}@ehu.eus Abstract Random walks over large knowledge bases like WordNet have been successfully used in word similarity, relatedness and disambigua- tion tasks. Unfortunately, those algorithms are relatively slow for large repositories, with significant memory footprints. In this pa- per we present a novel algorithm which en- codes the structure of a knowledge base in a continuous vector space, combining random walks and neural net language models in or- der to produce novel word representations. Evaluation in word relatedness and similar- ity datasets yields equal or better results than those of a random walk algorithm, using a dense representation (300 dimensions instead of 117K). Furthermore, the word representa- tions are complementary to those of the ran- dom walk algorithm and to corpus-based con- tinuous representations, improving the state- of-the-art in the similarity dataset. Our tech- nique opens up exciting opportunities to com- bine distributional and knowledge-based word representations. 1 Introduction Graph-based techniques over Knowledge Bases (KB) like WordNet (Fellbaum, 1998) have been widely used in NLP tasks, including word sense disambiguation (Agirre et al., 2014; Moro et al., 2014), semantic similarity and semantic relatedness between terms (Agirre et al., 2009; Agirre et al., 2010; Pilehvar et al., 2013). For instance, Agirre et al. (2009; 2010) apply a random walk algorithm based on Personalized PageRank to WordNet, pre- Figure 1: Main architecture for generating KB word embeddings. A random walk algorithm over the KB produces a synthetic corpus, which is fed into a NNLM to produce continuous word representations. senting the best results to date among WordNet- based methods for the well-known WS353 word- similarity dataset (Finkelstein et al., 2001). For each target word, the method performs a personal- ized random walk on the WordNet graph. At con- vergence, the target word is represented as a vector in a multi-dimensional conceptual space, with one dimension for each concept in the KB. The good results of the algorithm contrast with the large di- mensionality of the vectors that it needs to produce, 117K dimensions (one per synset) for WordNet. In recent years a wide variety of Neural Network Language Models (NNLM) have been successfully employed in several tasks, including word similarity (Collobert and Weston, 2008; Socher et al., 2011; 1434 Turian et al., 2010). NNLM extract meaning from unlabeled corpora following the distributional hy- pothesis (Harris, 1954), where semantic features of a word are related to its co-occurrence patterns. NNLM learn word representations in the form of dense scalar vectors in n-dimensional spaces (e.g. 300 dimensions), in which each dimension is a la- tent semantic feature. The representations are ob- tained by optimizing the likelihood of existing un- labeled text. More recently, Mikolov et al. …

Content truncated to 500 words