Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3308–3318 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3308 Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks Shikhar Vashishth1 Manik Bhandari1∗ Prateek Yadav2∗ Piyush Rai3 Chiranjib Bhattacharyya1 Partha Talukdar1 1Indian Institute of Science 2Microsoft Research, 3IIT Kanpur {shikhar,manikb,chiru,ppt}@iisc.ac.in t-pryad@microsoft.com, piyush@cse.iitk.ac.in Abstract Word embeddings have been widely adopted across several NLP applications. Most exist- ing word embedding methods utilize sequen- tial context of a word to learn its embedding. While there have been some attempts at uti- lizing syntactic context of a word, such meth- ods result in an explosion of the vocabulary size. In this paper, we overcome this prob- lem by proposing SynGCN, a flexible Graph Convolution based method for learning word embeddings. SynGCN utilizes the dependency context of a word without increasing the vo- cabulary size. Word embeddings learned by SynGCN outperform existing methods on var- ious intrinsic and extrinsic tasks and provide an advantage when used with ELMo. We also propose SemGCN, an effective framework for incorporating diverse semantic knowledge for further enhancing learned word representa- tions. We make the source code of both models available to encourage reproducible re- search. 1 Introduction Representing words as real-valued vectors is an effective and widely adopted technique in NLP. Such representations capture properties of words based on their usage and allow them to generalize across tasks. Meaningful word embeddings have been shown to improve performance on several relevant tasks, such as named entity recognition (NER) (Bengio et al., 2013), parsing (Socher et al., 2013), and part-of-speech (POS) tagging (Ma and Hovy, 2016). Using word embeddings for initial- izing Deep Neural Networks has also been found to be quite useful (Collobert et al., 2011; Johnson et al., 2017; Strubell et al., 2018). Most popular methods for learning word em- beddings are based on the distributional hypoth- esis, which utilizes the co-occurrence statistics ∗Contributed equally to the work. from sequential context of words for learning word representations (Mikolov et al., 2013a; Pen- nington et al., 2014). More recently, this approach has been extended to include syntactic contexts (Levy and Goldberg, 2014) derived from depen- dency parse of text. Higher order dependencies have also been exploited by Komninos and Man- andhar (2016); Li et al. (2018). Syntax-based embeddings encode functional similarity (in-place substitutable words) rather than topical similarity (topically related words) which provides an advan- tage on specific tasks like question classification (Komninos and Manandhar, 2016). However, cur- rent approaches incorporate syntactic context by concatenating words with their dependency rela- tions. For instance, in Figure 1 scientists_subj, water_obj, and mars_nmod needs to be included as a part of vocabulary for utilizing the depen- dency context of discover. This severely expands the vocabulary, thus limiting the scalability of models on large corpora. For instance, in Levy and Goldberg (2014) and Komninos and Man- andhar (2016), the context vocabulary explodes to around 1.3 million for learning embeddings of 220k words. Incorporating …

Content truncated to 500 words