Whom to Learn From? Graph- vs. Text-based Word Embeddings

Proceedings of Recent Advances in Natural Language Processing, pages 1041–1051, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_120 1041 Whom to Learn From? Graph- vs. Text-based Word Embeddings Małgorzata Salawa1,2, Ant´onio Branco1, Ruben Branco1, Jo˜ao Rodrigues1 and Chakaveh Saedi1,3 1University of Lisbon NLX-Natural Language and Speech Group, Faculdade de Ciˆencias, 1749-016 Lisboa, Portugal 2AGH University of Science and Technology Faculty of Computer Science, Electronics and Telecommunications, 30-001 Krak´ow, Poland 3Macquarie University Department of Computing, Sydney, NSW, 2109, Australia Abstract Vectorial representations of meaning can be supported by empirical data from diverse sources and obtained with diverse embedding approaches. This paper aims at screening this experimental space and reports on an assess- ment of word embeddings supported (i) by data in raw texts vs. in lexical graphs, (ii) by lexical information encoded in association- vs. inference-based graphs, and obtained (iii) by edge reconstruction- vs. matrix factori- sation vs. random walk-based graph embed- ding methods. The results observed with these experiments indicate that the best solutions with graph-based word embeddings are very competitive, consistently outperforming main- stream text-based ones. 1 Introduction As neural networks are becoming a central tech- nology in natural language processing, interest on distributional semantics, with its vector space models of meaning, has been a driving factor for research on natural language semantics. When fo- cusing on the meaning of words under this ap- proach, information on lexical semantics has been sought to be encoded into appropriate vectorial representations, also known as word embeddings. The source for this information has consisted mostly of large collections of raw text, and thus ultimately on the frequencies of co-occurrence of words with other neighbouring words in certain windows of context, (Mikolov et al., 2013a; Pen- nington et al., 2014; Mikolov et al., 2018) among others. A few research trends have been gaining momentum concerning the application of neural networks to natural language technology, and a fortiori in what concerns distributional semantics. On the one hand, there has been a growing inter- est in the linguistic information that may be ulti- mately encoded in vectorial representations (Be- linkov et al., 2017; Conneau et al., 2018), also relating to their eventual ”universality”, in view of possibly transferring these representations from one language processing task or application to an- other (Shi et al., 2016; C´ıfka and Bojar, 2018). On the other hand, growing attention has been devoted to sources of information for word embed- dings other than what may be conveyed and ex- tracted from co-occurrences in text. This includes information that is encoded in sophisticated lex- ical collections of data that are carefully crafted and densely loaded with accurate information on lexical semantics (Goikoetxea et al., 2015; Saedi et al., 2018). The results reported in the present paper lies at the intersection of those research goals. In par- ticular, we aim here to gain a better insight into these two sources of lexical information, and the quality of the resulting word embeddings, by as- sessing how graph-based word embeddings com- pare to mainstream text-based ones. To …

Content truncated to 500 words