Dict2vec : Learning Word Embeddings using Lexical Dictionaries

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 254–263 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Dict2vec : Learning Word Embeddings using Lexical Dictionaries Julien Tissier and Christophe Gravier and Amaury Habrard Univ. Lyon, UJM Saint-Etienne CNRS, Lab Hubert Curien UMR 5516 F-42023, Saint-Etienne, France {firstname.lastname}@univ-st-etienne.fr Abstract Learning word embeddings on large unla- beled corpus has been shown to be suc- cessful in improving many natural lan- guage tasks. The most efficient and pop- ular approaches learn or retrofit such rep- resentations using additional external data. Resulting embeddings are generally bet- ter than their corpus-only counterparts, al- though such resources cover a fraction of words in the vocabulary. In this paper, we propose a new approach, Dict2vec, based on one of the largest yet refined datasource for describing words – natural language dictionaries. Dict2vec builds new word pairs from dictionary entries so that semantically-related words are moved closer, and negative sampling filters out pairs whose words are unrelated in dictio- naries. We evaluate the word representa- tions obtained using Dict2vec on eleven datasets for the word similarity task and on four datasets for a text classification task. 1 Introduction Learning word embeddings usually relies on the distributional hypothesis – words appearing in similar contexts must have similar meanings, and thus close representations. Finding such represen- tations for words and sentences has been one hot topic over the last few years in Natural Language Processing (NLP) (Mikolov et al., 2013; Penning- ton et al., 2014) and has led to many improvements in core NLP tasks such as Word Sense Disam- biguation (Iacobacci et al., 2016), Machine Trans- lation (Devlin et al., 2014), Machine Comprehen- sion (Hewlett et al., 2016), and Semantic Role La- beling (Zhou and Xu, 2015; Collobert et al., 2011) – to name a few. These methods suffer from a classic drawback of unsupervised learning: the lack of supervision between a word and those appearing in the associ- ated contexts. Indeed, it is likely that some terms of the context are not related to the considered word. On the other hand, the fact that two words do not appear together – or more likely, not often enough together – in any context of the training corpora is not a guarantee that these words are not semantically related. Recent approaches have pro- posed to tackle this issue using an attentive model for context selection (Ling et al., 2015), or by us- ing external sources – like knowledge graphs – in order to improve the embeddings (Wang et al., 2014). Similarities derived from such resources are part of the objective function during the learn- ing phase (Yu and Dredze, 2014; Kiela et al., 2015) or used in a retrofitting scheme (Faruqui et al., 2015). These approaches tend to specialize the embeddings to the resource used and its asso- ciated similarity measures – while the construction and maintenance of these resources are a set of complex, time-consuming, and error-prone tasks. In this paper, we …

Content truncated to 500 words