Embedding Words and Senses Together via Joint Knowledge-Enhanced Training

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 100–111, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Embedding Words and Senses Together via Joint Knowledge-Enhanced Training Massimiliano Mancini*, Jose Camacho-Collados*, Ignacio Iacobacci and Roberto Navigli Department of Computer Science Sapienza University of Rome mancini@dis.uniroma1.it {collados,iacobacci,navigli}@di.uniroma1.it Abstract Word embeddings are widely used in Nat- ural Language Processing, mainly due to their success in capturing semantic infor- mation from massive corpora. However, their creation process does not allow the different meanings of a word to be auto- matically separated, as it conflates them into a single vector. We address this issue by proposing a new model which learns word and sense embeddings jointly. Our model exploits large corpora and knowl- edge from semantic networks in order to produce a unified vector space of word and sense embeddings. We evaluate the main features of our approach both qual- itatively and quantitatively in a variety of tasks, highlighting the advantages of the proposed method in comparison to state- of-the-art word- and sense-based models. 1 Introduction Recently, approaches based on neural networks which embed words into low-dimensional vector spaces from text corpora (i.e. word embeddings) have become increasingly popular (Mikolov et al., 2013; Pennington et al., 2014). Word embeddings have proved to be beneficial in many Natural Lan- guage Processing tasks, such as Machine Transla- tion (Zou et al., 2013), syntactic parsing (Weiss et al., 2015), and Question Answering (Bordes et al., 2014), to name a few. Despite their suc- cess in capturing semantic properties of words, these representations are generally hampered by an important limitation: the inability to discrimi- nate among different meanings of the same word. Authors marked with an asterisk (*) contributed equally. Previous works have addressed this limita- tion by automatically inducing word senses from monolingual corpora (Sch¨utze, 1998; Reisinger and Mooney, 2010; Huang et al., 2012; Di Marco and Navigli, 2013; Neelakantan et al., 2014; Tian et al., 2014; Li and Jurafsky, 2015; Vu and Parker, 2016; Qiu et al., 2016), or bilingual par- allel data (Guo et al., 2014; Ettinger et al., 2016; ˇSuster et al., 2016). However, these approaches learn solely on the basis of statistics extracted from text corpora and do not exploit knowl- edge from semantic networks. Additionally, their induced senses are neither readily interpretable (Panchenko et al., 2017) nor easily mappable to lexical resources, which limits their application. Recent approaches have utilized semantic net- works to inject knowledge into existing word rep- resentations (Yu and Dredze, 2014; Faruqui et al., 2015; Goikoetxea et al., 2015; Speer and Lowry- Duda, 2017; Mrksic et al., 2017), but without solv- ing the meaning conflation issue. In order to ob- tain a representation for each sense of a word, a number of approaches have leveraged lexical resources to learn sense embeddings as a result of post-processing conventional word embeddings (Chen et al., 2014; Johansson and Pina, 2015; Jauhar et al., 2015; Rothe and Sch¨utze, 2015; Pile- hvar and Collier, …

Content truncated to 500 words