attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines

Proceedings of NAACL-HLT 2018, pages 453–462 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics attr2vec: Jointly Learning Word and Contextual Attribute Embeddings with Factorization Machines Fabio Petroni Thomson Reuters Corporate Research & Development 30 South Colonnade, London, UK fabio.petroni@tr.com Vassilis Plachouras∗ Facebook 1 Rathbone Square, London, UK vplachouras@fb.com Timothy Nugent Thomson Reuters Corporate Research & Development 30 South Colonnade, London, UK tim.nugent@tr.com Jochen L. Leidner Thomson Reuters Corporate Research & Development 30 South Colonnade, London, UK jochen.leidner@tr.com Abstract The widespread use of word embeddings is associated with the recent successes of many natural language processing (NLP) systems. The key approach of popular models such as word2vec and GloVe is to learn dense vec- tor representations from the context of words. More recently, other approaches have been proposed that incorporate different types of contextual information, including topics, de- pendency relations, n-grams, and sentiment. However, these models typically integrate only limited additional contextual information, and often in ad hoc ways. In this work, we introduce attr2vec, a novel framework for jointly learning embeddings for words and contextual attributes based on fac- torization machines. We perform experiments with different types of contextual information. Our experimental results on a text classifica- tion task demonstrate that using attr2vec to jointly learn embeddings for words and Part- of-Speech (POS) tags improves results com- pared to learning the embeddings indepen- dently. Moreover, we use attr2vec to train dependency-based embeddings and we show that they exhibit higher similarity between functionally related words compared to tradi- tional approaches. 1 Introduction Neural network-based methods have been suc- cessful in advancing the state-of-the-art in a wide range of NLP tasks, such as depen- dency parsing (Chen and Manning, 2014), sen- tence classification (Kim, 2014), machine trans- lation (Sutskever et al., 2014; Luong and Man- ning, 2016), and information retrieval (Zhang ∗work conducted whilst the author was at Thomson Reuters. et al., 2017). In all these approaches, vecto- rial distributed word representations, known as word embeddings, have become a fundamental building block. The use of word embeddings is considered a “secret sauce” for contributing to the success of many of these algorithms in re- cent years (Luong et al., 2013). Popular mod- els for learning such word embeddings include word2vec (Mikolov et al., 2013a,b,c), GloVe (Pen- nington et al., 2014) and fastText (Bojanowski et al., 2017; Joulin et al., 2017). The main idea behind these techniques is to rep- resent a word by means of its context. The most popular forms of context are neighboring words in a window of text (Mikolov et al., 2013b; Penning- ton et al., 2014), though examples of additional contextual information might also include docu- ment topics (Li et al., 2016), dependency rela- tions (Levy and Goldberg, 2014), morphemes (Lu- ong et al., 2013), n-grams (Bojanowski et al., 2017), and sentiment (Tang et al., 2014). The em- bedding idea was originally devised to help over- come problems associated with the high dimen- sionality of sparse vector representations of words, particularly in the …

Content truncated to 500 words