MUSE: Modularizing Unsupervised Sense Embeddings

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 327–337 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics MUSE: Modularizing Unsupervised Sense Embeddings Guang-He Lee⋆† Yun-Nung Chen† ⋆Massachusetts Institute of Technology, Cambridge, MA †National Taiwan University, Taipei, Taiwan guanghe@mit.edu y.v.chen@ieee.org Abstract This paper proposes to address the word sense ambiguity issue in an unsupervised manner, where word sense representations are learned along a word sense selection mechanism given contexts. Prior work focused on designing a single model to deliver both mechanisms, and thus suf- fered from either coarse-grained represen- tation learning or inefficient sense selec- tion. The proposed modular approach, MUSE, implements flexible modules to optimize distinct mechanisms, achieving the first purely sense-level representation learning system with linear-time sense se- lection. We leverage reinforcement learn- ing to enable joint training on the pro- posed modules, and introduce various ex- ploration techniques on sense selection for better robustness. The experiments on benchmark data show that the proposed approach achieves the state-of-the-art per- formance on synonym selection as well as on contextual word similarities in terms of MaxSimC. 1 Introduction Recently, deep learning methodologies have dom- inated several research areas in natural language processing (NLP), such as machine translation, language understanding, and dialogue systems. However, most of applications usually utilize word-level embeddings to obtain semantics. Con- sidering that natural language is highly ambigu- ous, the standard word embeddings may suffer from polysemy issues. Neelakantan et al. (2014) pointed out that, due to triangle inequality in vec- tor space, if one word has two different senses but is restricted to one embedding, the sum of the distances between the word and its synonym in each sense would upper-bound the distance be- tween the respective synonyms, which may be mu- tually irrelevant, in embedding space1. Due to the theoretical inability to account for polysemy us- ing a single embedding representation per word, multi-sense word representations are proposed to address the ambiguity issue using multiple em- bedding representations for different senses in a word (Reisinger and Mooney, 2010; Huang et al., 2012). This paper focuses on unsupervised learning from the unannotated corpus. There are two key mechanisms for a multi-sense word representation system in such scenario: 1) a sense selection (de- coding) mechanism infers the most probable sense for a word given its context and 2) a sense repre- sentation mechanism learns to embed word senses in a continuous space. Under this framework, prior work focused on designing a single model to deliver both mech- anisms (Neelakantan et al., 2014; Li and Juraf- sky, 2015; Qiu et al., 2016). However, the previ- ously proposed models introduce side-effects: 1) mixing word-level and sense-level tokens achieves efficient sense selection but introduces ambigu- ous word-level tokens during the representation learning process (Neelakantan et al., 2014; Li and Jurafsky, 2015), and 2) pure sense-level tokens prevent ambiguity from word-level tokens but re- quire exponential time complexity when decoding a sense sequence (Qiu et al., 2016). Unlike the prior work, this paper proposes MUSE2—a …

Content truncated to 500 words