The Gavagai Living Lexicon

The Gavagai Living Lexicon Magnus Sahlgren†, Amaru Cuba Gyllensten†, Fredrik Espinoza†, Ola Hamfors†, Jussi Karlgren† Fredrik Olsson†, Per Persson†, Akshay Viswanathan† and Anders Holst⋆ †Gavagai, Slussplan 9, 111 30 Stockholm, Sweden ⋆SICS, Box 1263, 164 29 Kista, Sweden [firstname.lastname]@gavagai.se, aho@sics.se Abstract This paper presents the Gavagai Living Lexicon, which is an online distributional semantic model currently available in 20 different languages. We describe the underlying distributional semantic model, and how we have solved some of the challenges in applying such a model to large amounts of streaming data. We also describe the architecture of our implementation, and discuss how we deal with continuous quality assurance of the lexicon. Keywords: Distributional semantics, word embeddings, lexical resources 1. Introduction The availability of large amounts of text data in open sources like online news and social media, coupled with the recent developments of scalable and effective techniques for computational semantics, opens up new possibilities for lexicographic research and resource development. As a complement to traditional lexica and thesauri, it is now possible to automatically build lexical resources by min- ing large amounts of text data using unsupervised machine learning methods. The perhaps most well-known example of a data-driven thesaurus is the Sketch Engine,1 which features distribu- tional thesauri for some 60 different languages (Kilgar- riff et al., 2014). Another example is the polyglot Python library,2 which contains word embeddings trained on Wikipedia for 137 different languages (Al-Rfou et al., 2013). There are also several academic projects that pro- vide data-driven thesauri, like the Wortschatz project,3 and the JoBimText project.4 This paper presents a continuously learning distributional thesaurus — the Gavagai Living Lexicon — that updates its semantic model according to the input data stream in an online fashion, and produces lexical entries in several different languages.5 The lexicon currently includes sev- eral different types of entries: string similar terms, topically related terms, sequentially related terms, multiword terms, and semantically similar terms. In the following sections, we describe how we use an online distributional semantic model to identify and compile these various relations. We also describe the architecture of our implementation, and discuss how we deal with continuous quality assurance. 1www.sketchengine.co.uk 2pypi.python.org/pypi/polyglot 3corpora.informatik.uni-leipzig.de 4maggie.lt.informatik.tu-darmstadt.de/jobimtext 5As of 2016-03-10, the lexicon includes the following 20 languages: Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Hebrew, Hungarian, Italian, Latvian, Lithua- nian, Norwegian, Polish, Portuguese, Romanian, Russian, Span- ish, Swedish. 2. (Online) Distributional Semantics Distributional Semantic Models (DSMs) represent terms as distributional vectors that record (some function of) co- occurrence counts collected within a context window sur- rounding each occurrence of the terms. There are many different approaches to building such vectors, ranging from simple accumulation of co-occurrence frequencies to more advanced methods based on matrix factorization ( ¨Osterlund et al., 2015) and artificial neural networks (Turian et al., 2010). The particular framework we use for building DSMs is called Random Indexing (RI) (Kanerva et al., 2000; Kan- erva, 2009), which provides an attractive processing model for streaming environments because it can be formulated as an online model …

Content truncated to 500 words