Better Word Embeddings by Disentangling Contextual n-Gram Information

Proceedings of NAACL-HLT 2019, pages 933–939 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 933 Better Word Embeddings by Disentangling Contextual n-Gram Information Prakhar Gupta* EPFL Switzerland prakhar.gupta@epfl.ch Matteo Pagliardini* Iprova SA, Switzerland mpagliardini@iprova.com Martin Jaggi EPFL Switzerland martin.jaggi@epfl.ch Abstract Pre-trained word vectors are ubiquitous in Natural Language Processing applications. In this paper, we show how training word em- beddings jointly with bigram and even trigram embeddings, results in improved unigram em- beddings. We claim that training word embed- dings along with higher n-gram embeddings helps in the removal of the contextual infor- mation from the unigrams, resulting in better stand-alone word embeddings. We empirically show the validity of our hypothesis by outper- forming other competing word representation models by a significant margin on a wide va- riety of tasks. We make our models publicly available. 1 Introduction Distributed word representations are essential building blocks of modern NLP systems. Used as features in downstream applications, they often enhance generalization of models trained on a lim- ited amount of data. They do so by capturing rel- evant distributional information about words from large volumes of unlabeled text. Efficient methods to learn word vectors have been introduced in the past, most of them based on the distributional hypothesis of Harris (1954); Firth (1957): “a word is characterized by the com- pany it keeps”. While a standard approach re- lies on global corpus statistics (Pennington et al., 2014) formulated as a matrix factorization us- ing mean square reconstruction loss, other widely used methods are the bilinear word2vec architec- tures introduced by Mikolov et al. (2013a): While skip-gram aims to predict nearby words from a given word, CBOW predicts a target word from its set of context words. Recently, significant improvements in the qual- ity of the word embeddings were obtained by * indicates equal contribution augmenting word-context pairs with sub-word in- formation in the form of character n-grams (Bo- janowski et al., 2017), especially for morpholog- ically rich languages. Nevertheless, to the best of our knowledge, no method has been introduced leveraging collocations of words with higher order word n-grams such as bigrams or trigrams as well as character n-grams together. In this paper, we show how using higher order word n-grams along with unigrams during training can significantly improve the quality of obtained word embeddings. The addition furthermore helps to disentangle contextual information present in the training data from the unigrams and results in overall better distributed word representations. To validate our claim, we train two modifica- tions of CBOW augmented with word-n-gram in- formation during training. One is a recent sen- tence embedding method, Sent2Vec (Pagliardini et al., 2018), which we repurpose to obtain word vectors. The second method we propose is a modification of CBOW enriched with character n- gram information (Bojanowski et al., 2017) that we again augment with word n-gram information. In both cases, we compare the resulting vectors with the most widely used word embedding meth- ods on word similarity and analogy …

Content truncated to 500 words