Projection Aléatoire Non-Négative pour le Calcul de Word Embedding / Non-Negative Randomized Word Embedding

Projection Aléatoire Non-Négative pour le Calcul de Word Embedding Behrang Q. Zadeh1 Laura Kallmeyer1 Aurelie Herbelot2 (1) DFG SFB 991, Universität Düsseldorf, Düsseldorf, Germany (2) CLIC-CIMEC, University of Trento, Italy zadeh@phil.hhu.de., kallmeyer@phil.hhu.de, aurelie.herbelot@cantab.net RÉSUMÉ Dans cet article nous proposons une nouvelle méthode pour le calcul de word embeddings en utilisant des projections aléatoires. Notre approche est telle que des méthodes de pondération comme positive pointwise mutual information (PPMI) peuvent être appliquées après la construction de notre modèle véctoriel, i.e., après avoir déjà réduit la dimensionalité. Les word embeddings peuvent alors être tranferés d’une manière efficace et avec une performance de calcul supérieure vers des espaces sémantiquement discriminants. De plus, l’approche se distingue par une mise-à-jour facilitée de l’espace vectoriel et une interopérabilité augmentée. Nous évaluons notre méthode par rapport à plusieurs tâches sémantiques. Nous montrons qu’elle donne des résultats comparables à ceux des meilleures méthodes appliquées à des corpus monolingues. ABSTRACT Non-Negative Randomized Word Embedding We propose a word embedding method which is based on a novel random projection technique. We show that weighting methods such as positive pointwise mutual information (PPMI) can be applied to our models after their construction and at a reduced dimensionality. Hence, the proposed technique can efficiently transfer words onto semantically discriminative spaces while demonstrating high computational performance, besides benefits such as ease of update and a simple mechanism for interoperability. We report the performance of our method on several tasks and show that it yields competitive results compared to neural embedding methods in monolingual corpus-based setups. MOTS-CLÉS : Word Embedding, Projection Aléatoire. KEYWORDS: Word Embedding, Word Vectors, Random Projections, Lexical Semantics. 1 Introduction Word embedding techniques have become a cornerstone of several modern NLP systems. From a methodological perspective, two broad categories of embedding algorithms are commonly identified and used. Following the terminology proposed in Baroni et al. (2014), these are A) the classic count models, which employ the chain of processes of collecting co-occurrence frequencies in high- dimensional vectors followed by weighting and dimensionality reduction processes (see Turney & Pantel, 2010) ; and, B) the more recent neural predictive techniques in which the process of learning word embeddings is modelled as a (type of convex) optimisation task, and implemented using a kind of neural network—e.g., Mikolov et al. (2013b), Pennington et al. (2014), and so on. Evidently, both methods capture statistical information about word usage and can be explained using vector space mathematics (Levy & Goldberg, 2014). In this paper, we ask which features an ideal vector-based representation of word meaning should have. We maintain that an ideal model must A) show human-like behaviour on linguistic tasks of interest ; B) have low dimensionality for ‘easy’ manipulation ; C) be efficiently acquirable from large corpora to allow for fast experimentation ; D) be linked to a transparent model, so that linguistic and computational hypotheses and experimental results can be systematically analysed and explained ; and E) be easy to update (e.g., allow the addition of new context elements or target entities and the …

Content truncated to 500 words