Semantic Specialization of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints

Semantic Specialization of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints Nikola Mrkši´c1,2, Ivan Vuli´c1, Diarmuid Ó Séaghdha2, Ira Leviant3 Roi Reichart3, Milica Gaši´c1, Anna Korhonen1, Steve Young1,2 1 University of Cambridge 2 Apple Inc. 3 Technion, IIT Abstract We present ATTRACT-REPEL, an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources. ATTRACT-REPEL facilitates the use of constraints from mono- and cross- lingual resources, yielding semantically spe- cialized cross-lingual vector spaces. Our evalu- ation shows that the method can make use of ex- isting cross-lingual lexicons to construct high- quality vector spaces for a plethora of different languages, facilitating semantic transfer from high- to lower-resource ones. The effectiveness of our approach is demonstrated with state-of- the-art results on semantic similarity datasets in six languages. We next show that ATTRACT- REPEL-specialized vectors boost performance in the downstream task of dialogue state track- ing (DST) across multiple languages. Finally, we show that cross-lingual vector spaces pro- duced by our algorithm facilitate the training of multilingual DST models, which brings further performance improvements. 1 Introduction Word representation learning has become a re- search area of central importance in modern natu- ral language processing. The common techniques for inducing distributed word representations are grounded in the distributional hypothesis, relying on co-occurrence information in large textual corpora to learn meaningful word representations (Mikolov et al., 2013b; Pennington et al., 2014; Ó Séaghdha and Korhonen, 2014; Levy and Goldberg, 2014). Re- cently, methods that go beyond stand-alone unsu- pervised learning have gained increased popularity. These models typically build on distributional ones by using human- or automatically-constructed knowl- edge bases to enrich the semantic content of existing word vector collections. Often this is done as a post- processing step, where the distributional word vectors are refined to satisfy constraints extracted from a lex- ical resource such as WordNet (Faruqui et al., 2015; Wieting et al., 2015; Mrkši´c et al., 2016). We term this approach semantic specialization. In this paper we advance the semantic specializa- tion paradigm in a number of ways. We introduce a new algorithm, ATTRACT-REPEL, that uses syn- onymy and antonymy constraints drawn from lexi- cal resources to tune word vector spaces using lin- guistic information that is difficult to capture with conventional distributional training. Our evaluation shows that ATTRACT-REPEL outperforms previous methods which make use of similar lexical resources, achieving state-of-the-art results on two word sim- ilarity datasets: SimLex-999 (Hill et al., 2015) and SimVerb-3500 (Gerz et al., 2016). We then deploy the ATTRACT-REPEL algorithm in a multilingual setting, using semantic relations ex- tracted from BabelNet (Navigli and Ponzetto, 2012; Ehrmann et al., 2014), a cross-lingual lexical re- source, to inject constraints between words of differ- ent languages into the word representations. This al- lows us to embed vector spaces of multiple languages into a single vector space, exploiting information from high-resource languages to improve the word representations of lower-resource ones. Table 1 illus- trates the effects of cross-lingual ATTRACT-REPEL specialization by showing the nearest neighbors …

Content truncated to 500 words