Training Word Sense Embeddings With Lexicon-based Regularization

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 284–294, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Training Word Sense Embeddings With Lexicon-based Regularization Luis Nieto-Pi˜na and Richard Johansson University of Gothenburg {luis.nieto.pina, richard.johansson}@gu.se Abstract We propose to improve word sense embeddings by enriching an automatic corpus-based method with lexicographic data. Information from a lexicon is intro- duced into the learning algorithm’s objec- tive function through a regularizer. The incorporation of lexicographic data yields embeddings that are able to reflect expert- defined word senses, while retaining the robustness, high quality, and coverage of automatic corpus-based methods. These properties are observed in a manual in- spection of the semantic clusters that dif- ferent degrees of regularizer strength cre- ate in the vector space. Moreover, we evaluate the sense embeddings in two downstream applications: word sense dis- ambiguation and semantic frame predic- tion, where they outperform simpler ap- proaches. Our results show that a corpus- based model balanced with lexicographic data learns better representations and im- prove their performance in downstream tasks. 1 Introduction Word embeddings, as a tool for representing the meaning of words based on the context in which they appear, have had a considerable impact on many of the traditional Natural Language Process- ing tasks in recent years. (Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011; Glo- rot et al., 2011) This form of semantic representa- tion has come to replace in many instances tradi- tional count-based vectors (Baroni et al., 2014), as they yield high-quality semantic representations in a computationally efficient manner, which allows them to leverage information from large corpora. Due to this success, some attention has been de- voted to the question of whether their represen- tational power can be refined to further advance the state of the art in those tasks that can bene- fit from semantic representations. One instance in which this could be realized concerns polysemous words, which has led to several attempts at repre- senting word senses instead of simple word forms. Doing so would help avoid the situation in which several meanings of a word have to be conflated into just one embedding, typical of simple word embeddings. Among the different approaches to learning word sense embeddings, a distinction can be made between those that make use of a semantic net- work (SN) and those that do not. Approaches in the latter group usually apply an unsupervised strategy for clustering instances of words based on the context formed by surrounding words. The re- sulting clusters are then used to represent the dif- ferent meanings of a word. These representations characterize word usage in the training corpus rather than lexicographic senses, and run the risk of marginalizing under-represented word senses. Nonetheless, for well represented word senses, this strategy proves to be effective and adaptable to changes. The alternative is to integrate an SN in the learn- ing process. This kind of resource encodes a lex- icon of word senses, connecting lexically …

Content truncated to 500 words