Crossmodal Network-Based Distributional Semantic Models

Crossmodal Network-Based Distributional Semantic Models Elias Iosif and Alexandros Potamianos School of Electrical & Computer Engineering, National Technical University of Athens, Greece “Athena” Research and Innovation Center, Greece iosife@central.ntua.gr, potam@central.ntua.gr Abstract Despite the recent success of distributional semantic models (DSMs) in various semantic tasks they remain disconnected with real-world perceptual cues since they typically rely on linguistic features. Text data constitute the dominant source of features for the majority of such models, although there is evidence from cognitive science that cues from other modalities contribute to the acquisition and representation of semantic knowledge. In this work, we propose the crossmodal extension of a two-tier text-based model, where semantic representations are encoded in the first layer, while the second layer is used for computing similarity between words. We exploit text- and image-derived features for performing computations at each layer, as well as various approaches for their crossmodal fusion. It is shown that the crossmodal model performs better (from 0.68 to 0.71 correlation coefficient) than the unimodal one for the task of similarity computation between words. Keywords: activation based models, crossmodal fusion, distributional semantic models, semantic similarity, bag-of-visual-words 1. Introduction Distributional semantic models (DSMs) (Baroni and Lenci, 2010) constitute a widely-used paradigm for ex- tracting, representing and learning semantics from lin- guistic data. DSMs are based on the distributional hy- pothesis of meaning (Harris, 1954) assuming that seman- tic similarity between words is a function of the overlap of their linguistic contexts. Despite their success in vari- ous semantic tasks (e.g., semantic classification and com- putation of semantic similarity) these models have been criticized as “disembodied”, since they rely solely on lin- guistic information without being grounded in perception and action including other modalities, e.g., color (Barsa- lou et al., 2008). This disconnection is also referred to as the symbol grounding problem (Harnad, 1990). Ground- ing refers to the mapping of the semantics of natural lan- guage to the physical world. This is supported by exper- imental findings indicating that real-world experiences also play a role for the acquisition of lexical semantics (Landau et al., 1998). For example, the naming of ob- jects from pictures was found to be faster for color im- ages vs grayscale (Therriault et al., 2009). Recently, fo- cus has been given to the incorporation of features from modalities other than text in order to augment the text- based DSMs, e.g., see (Bruni et al., 2014) for image- derived features, and for audio-based features (Kiela and Clark, 2015). For additional insight into computation models that map data-derived low-level features to high- level knowledge (including cognitive and social aspects) see (Potamianos, 2014). The proposed approach is an alternative framework for integrating textual and visual features for the task of se- mantic similarity computation between words. This is motivated by the cognitive evidence indicating the multi- modal character of semantic representations utilized for various semantic tasks. This work, extends the unimodal (lexical-only) approach of network-based DSMs that has been successfully applied for the task of word similarity achieving state-of-the-art results …

Content truncated to 500 words