Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding

Proceedings of COLING 2012: Technical Papers, pages 1933–1950, COLING 2012, Mumbai, December 2012. Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding Brian Murphy Partha Pratim Talukdar Machine Learning DepartmentTom Mitchell Carnegie Mellon University {bmurphy,ppt,tom}@cs.cmu.edu Abstract In this paper, we introduce an application of matrix factorization to produce corpus-derived, distribu- tional models of semantics that demonstrate cognitive plausibility. We find that word representations learned by Non-Negative Sparse Embedding (NNSE), a variant of matrix factorization, are sparse, effective, and highly interpretable. To the best of our knowledge, this is the first approach which yields semantic representation of words satisfying these three desirable properties. Though extensive experimental evaluations on multiple real-world tasks and datasets, we demonstrate the superiority of semantic models learned by NNSE over other state-of-the-art baselines. Keywords: distributional semantics, sparse coding, neuro-semantics, vector-space models, inter- pretability, word embeddings. 1933 1 Introduction State-of-the-art distributional models of semantics, also termed vector-space models or word em- beddings, derive word-representations in an unsupervised fashion from large corpora. They are primarily based on observed co-occurrence patterns, but are typically subsequently reduced in dimensionality using techniques such as Clustering, Latent Dirichlet Allocation (LDA) (Blei et al., 2003), or Singular Value Decomposition (SVD). They have proven effective as components of a wide range of NLP applications, and in the modelling of cognitive operations such as judgements of word similarity (Sahlgren, 2006; Turney and Pantel, 2010; Baroni and Lenci, 2010), and the brain activity elicited by particular concepts (Mitchell et al., 2008). However, with few exceptions (e.g. Baroni et al., 2010), the representations they derive from corpora are lacking in cognitive plausibility. For instance, one of the SVD-based models described in this paper models similarity very success- fully, revealing the set of words mango, plum, cranberry, blueberry, melon as the cosine-distance nearest neighbours of pear. However, the latent SVD dimension for which pear has its largest weighting is hard to interpret – its most strongly positively associated tokens are action, records, government, record, search, and negatively associated tokens are sound, need, s, species, award. In addition, the representation of pear is a dense vector of small positive or negative values on several hundred dimensions which are also active for words of all types and domains, whether semantically similar (e.g., other living things, and concrete objects) or dissimilar (e.g., abstract nominals, and function and content words of other parts-of-speech). Certain cognitive arguments against such a representation are based on economy of storage (see Schunn, 1999; Murphy, 2004; Griffiths et al., 2007 for broader discussion on this point). A-priori it seems unlikely that the same compact set of features are sufficient and necessary to describe all semantic domains of a full adult vocabulary. Some very specific words may need more detailed descriptions (i.e., more properties) and generic words less. Properties are restricted to particular classes of concepts – limbs to animals, functions to manipulable artefacts, wheels to vehicles – and even within restricted semantic domains some features can be very specific to certain concepts, such as screens to electronic …

Content truncated to 500 words