Using Sparse Semantic Embeddings Learned from Multimodal Text and Image Data to Model Human Conceptual Knowledge

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 260–270 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 260 Using sparse semantic embeddings learned from multimodal text and image data to model human conceptual knowledge Steven Derby1 Paul Miller1 Brian Murphy1,2 Barry Devereux1 1 Queen’s University Belfast, Belfast, United Kingdom 2 BrainWaveBank Ltd., Belfast, United Kingdom {sderby02, p.miller, brian.murphy, b.devereux}@qub.ac.uk Abstract Distributional models provide a convenient way to model semantics using dense embed- ding spaces derived from unsupervised learn- ing algorithms. However, the dimensions of dense embedding spaces are not designed to resemble human semantic knowledge. More- over, embeddings are often built from a sin- gle source of information (typically text data), even though neurocognitive research suggests that semantics is deeply linked to both lan- guage and perception. In this paper, we com- bine multimodal information from both text and image-based representations derived from state-of-the-art distributional models to pro- duce sparse, interpretable vectors using Joint Non-Negative Sparse Embedding. Through in- depth analyses comparing these sparse models to human-derived behavioural and neuroimag- ing data, we demonstrate their ability to pre- dict interpretable linguistic descriptions of hu- man ground-truth semantic knowledge. 1 Introduction Distributional Semantic Models (DSMs) are used to represent semantic information about concepts in a high-dimensional vector space, where each concept is represented as a point in the space such that concepts with more similar meanings are closer together. Unsupervised learning algorithms are regularly employed to produce these models, where learning depends on statistical regularities in the distribution of words, exploiting a theory in linguistics called the distributional hypothe- sis. Recent developments in deep learning have resulted in weakly-supervised prediction-based methods, where, for example, a neural network is trained to predict words from surrounding con- texts, and the network parameters are interpreted as vectors of the distributional model (Mikolov et al., 2013). Like their counterparts in machine vision, neural network algorithms for DSMs au- tomate feature extraction from highly complex data without prior feature selection (Krizhevsky et al., 2012; Mikolov et al., 2013; Karpathy and Li, 2015; Antol et al., 2015). Such deep learn- ing techniques have led to state-of-the-art perfor- mance in many domains, though this is often at the expense of the interpretability and cognitive plausibility of the learned features (Murphy et al., 2012; Zeiler and Fergus, 2013). Furthermore, these compact, dense embeddings are structurally dissimilar to the way in which humans conceptu- alise the meanings of words (McRae et al., 2005). One way of drawing interpretability from highly latent data is by transforming it into a sparse repre- sentation (Faruqui et al., 2015; Senel et al., 2017). Moreover, the design of distributional models has been for the most part unimodal, typically relying on text corpora, even though studies in psychol- ogy have shown that human semantic processing is deeply linked with visual perception. In cognitive neuroscience, research demon- strates that representations of high-level concepts corresponding to the meanings of nouns and vi- sual objects are widely distributed and …

Content truncated to 500 words