Of Words, Eyes and Brains: Correlating Image-Based Distributional Semantic Models with Neural Representations of Concepts

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1960–1970, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Of words, eyes and brains: Correlating image-based distributional semantic models with neural representations of concepts Andrew J. Anderson, Elia Bruni, Ulisse Bordignon, Massimo Poesio and Marco Baroni Center for Mind/Brain Sciences (University of Trento, C.so Bettini 31, Rovereto, Italy) first.last@unitn.it Abstract Traditional distributional semantic models ex- tract word meaning representations from co- occurrence patterns of words in text cor- pora. Recently, the distributional approach has been extended to models that record the co- occurrence of words with visual features in image collections. These image-based models should be complementary to text-based ones, providing a more cognitively plausible view of meaning grounded in visual perception. In this study, we test whether image-based mod- els capture the semantic patterns that emerge from fMRI recordings of the neural signal. Our results indicate that, indeed, there is a significant correlation between image-based and brain-based semantic similarities, and that image-based models complement text-based ones, so that the best correlations are achieved when the two modalities are combined. De- spite some unsatisfactory, but explained out- comes (in particular, failure to detect differ- ential association of models with brain areas), the results show, on the one hand, that image- based distributional semantic models can be a precious new tool to explore semantic repre- sentation in the brain, and, on the other, that neural data can be used as the ultimate test set to validate artificial semantic models in terms of their cognitive plausibility. 1 Introduction Many recent neuroscientific studies have brought support to the view that concepts are represented in terms of patterns of neural activation over broad areas, naturally encoded as vectors in a neural se- mantic space (Haxby et al., 2001; Huth et al., 2012). Similar representations are also widely used in com- putational linguistics, and in particular in distribu- tional semantics (Clark, 2012; Erk, 2012; Turney and Pantel, 2010), that captures meaning in terms of vectors recording the patterns of co-occurrence of words in large corpora, under the hypothesis that words that occur in similar contexts are similar in meaning. Since the seminal work of Mitchell et al. (2008), there has thus being interest in investigating whether corpus-harvested semantic representations can con- tribute to the study of concepts in the brain. The relation is mutually beneficial: From the point of view of brain activity decoding, a strong correlation between corpus-based and brain-derived conceptual representations would mean that we could use the former (much easier to construct on a very large scale) to make inferences about the second: e.g., us- ing corpus-based representations to reconstruct the likely neural signal associated to words we have no direct brain data for. From the point of view of com- putational linguistics, neural data provide the ulti- mate testing ground for models that strive to cap- ture important aspects of human semantic mem- ory (much more so than the commonly used ex- plicit semantic rating …

Content truncated to 500 words