LSTMEmbed: Learning Word and Sense Representations from a Large Semantically Annotated Corpus with Long Short-Term Memories

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1685–1695 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1685 LSTMEmbed: Learning Word and Sense Representations from a Large Semantically Annotated Corpus with Long Short-Term Memories Ignacio Iacobacci1,2∗and Roberto Navigli2 1Huawei Noah’s Ark Lab, London, United Kingdom 2Department of Computer Science, Sapienza University of Rome, Italy ignacio.iacobacci@huawei.com {iacobacci,navigli}@di.uniroma1.it Abstract While word embeddings are now a de facto standard representation of words in most NLP tasks, recently the attention has been shifting towards vector representations which capture the different meanings, i.e., senses, of words. In this paper we explore the capabilities of a bidirectional LSTM model to learn represen- tations of word senses from semantically an- notated corpora. We show that the utilization of an architecture that is aware of word or- der, like an LSTM, enables us to create bet- ter representations. We assess our proposed model on various standard benchmarks for evaluating semantic representations, reaching state-of-the-art performance on the SemEval- 2014 word-to-sense similarity task. We re- lease the code and the resulting word and sense embeddings at http://lcl.uniroma1. it/LSTMEmbed. 1 Introduction Natural Language is inherently ambiguous, for reasons of communicative efficiency (Piantadosi et al., 2012). For us humans, ambiguity is not a problem, since we use common knowledge to fill in the gaps and understand each other. There- fore, a computational model suited to understand- ing natural language and working side by side with humans should be capable of dealing with ambi- guity to a certain extent (Navigli, 2018). A neces- sary step towards creating such computer systems is to build formal representations of words and their meanings, either in the form of large reposi- tories of knowledge, e.g., semantic networks, or as vectors in a geometric space (Navigli and Martelli, 2019). In fact, Representation Learning (Bengio et al., 2013) has been a major research area in NLP over ∗ Ignacio Iacobacci’s work was mainly done at the Sapienza University of Rome. the years, and latent vector-based representations, called embeddings, seem to be a good candidate for coping with ambiguity. Embeddings encode lexical and semantic items in a low-dimensional continuous space. These vector representations capture useful syntactic and semantic information of words and senses, such as regularities in the nat- ural language, and relationships between them, in the form of relation-specific vector offsets. Recent approaches, such as word2vec (Mikolov et al., 2013), and GloVe (Pennington et al., 2014), are ca- pable of learning efficient word embeddings from large unannotated corpora. But while word em- beddings have paved the way for improvements in numerous NLP tasks (Goldberg, 2017), they still conflate the various meanings of each word and let its predominant sense prevail over all others in the resulting representation. Instead, when these em- bedding learning approaches are applied to sense- annotated data, they are able to produce embed- dings for word senses (Iacobacci et al., 2015). A strand of work aimed at tackling the lexi- cal polysemy issue has proposed the …

Content truncated to 500 words