Can Network Embedding of Distributional Thesaurus Be Combined with Word Vectors for Better Representation?

Proceedings of NAACL-HLT 2018, pages 463–473 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Can Network Embedding of Distributional Thesaurus be Combined with Word Vectors for Better Representation? Abhik Jana IIT Kharagpur Kharagpur, India abhik.jana@iitkgp.ac.in Pawan Goyal IIT Kharagpur Kharagpur, India pawang@cse.iitkgp.ac.in Abstract Distributed representations of words learned from text have proved to be successful in var- ious natural language processing tasks in re- cent times. While some methods represent words as vectors computed from text using predictive model (Word2vec) or dense count based model (GloVe), others attempt to rep- resent these in a distributional thesaurus net- work structure where the neighborhood of a word is a set of words having adequate con- text overlap. Being motivated by recent surge of research in network embedding techniques (DeepWalk, LINE, node2vec etc.), we turn a distributional thesaurus network into dense word vectors and investigate the usefulness of distributional thesaurus embedding in im- proving overall word representation. This is the first attempt where we show that combin- ing the proposed word representation obtained by distributional thesaurus embedding with the state-of-the-art word representations helps in improving the performance by a significant margin when evaluated against NLP tasks like word similarity and relatedness, synonym de- tection, analogy detection. Additionally, we show that even without using any handcrafted lexical resources we can come up with repre- sentations having comparable performance in the word similarity and relatedness tasks com- pared to the representations where a lexical re- source has been used. 1 Introduction Natural language understanding has always been a primary challenge in natural language process- ing (NLP) domain. Learning word representa- tions is one of the basic and primary steps in un- derstanding text and nowadays there are predomi- nantly two views of learning word representations. In one realm of representation, words are vectors of distributions obtained from analyzing their con- texts in the text and two words are considered meaningfully similar if the vectors of those words are close in the euclidean space. In recent times, attempts have been made for dense representa- tion of words, be it using predictive model like Word2vec (Mikolov et al., 2013) or count-based model like GloVe (Pennington et al., 2014) which are computationally efficient as well. Another stream of representation talks about network like structure where two words are considered neigh- bors if they both occur in the same context above a certain number of times. The words are finally represented using these neighbors. Distributional Thesaurus is one such instance of this type, which gets automatically produced from a text corpus and identifies words that occur in similar contexts; the notion of which was used in early work about distributional semantics (Grefenstette, 2012; Lin, 1998; Curran and Moens, 2002). One such repre- sentation is JoBimText proposed by Biemann and Riedl (2013) that contains, for each word, a list of words that are similar with respect to their bi- gram distribution, thus producing a network rep- resentation. Later, Riedl and Biemann (2013) in- troduced a …

Content truncated to 500 words