Visually Grounded and Textual Semantic Models Differentially Decode Brain Activity Associated with Concrete and Abstract Nouns

Visually Grounded and Textual Semantic Models Differentially Decode Brain Activity Associated with Concrete and Abstract Nouns Andrew J. Anderson Brain & Cognitive Sciences University of Rochester aander41@ur.rochester.edu Douwe Kiela Computer Laboratory University of Cambridge dk427@cam.ac.uk Stephen Clark Computer Laboratory University of Cambridge sc609@cam.ac.uk Massimo Poesio School of Computer Science and Electronic Engineering University of Essex poesio@essex.ac.uk Abstract Important advances have recently been made using computational semantic models to de- code brain activity patterns associated with concepts; however, this work has almost ex- clusively focused on concrete nouns. How well these models extend to decoding abstract nouns is largely unknown. We address this question by applying state-of-the-art compu- tational models to decode functional Magnetic Resonance Imaging (fMRI) activity patterns, elicited by participants reading and imagin- ing a diverse set of both concrete and abstract nouns. One of the models we use is linguistic, exploiting the recent word2vec skipgram ap- proach trained on Wikipedia. The second is visually grounded, using deep convolutional neural networks trained on Google Images. Dual coding theory considers concrete con- cepts to be encoded in the brain both linguisti- cally and visually, and abstract concepts only linguistically. Splitting the fMRI data accord- ing to human concreteness ratings, we indeed observe that both models significantly decode the most concrete nouns; however, accuracy is significantly greater using the text-based mod- els for the most abstract nouns. More gener- ally this confirms that current computational models are sufficiently advanced to assist in investigating the representational structure of abstract concepts in the brain. 1 Introduction Since the work of Mitchell et al. (2008), there has been increasing interest in using computational se- mantic models to interpret neural activity patterns scanned as participants engage in conceptual tasks. This research has almost exclusively focused on brain activity elicited as participants comprehend concrete nouns as experimental stimuli. Different modelling approaches — predominantly distribu- tional semantic models (Mitchell et al., 2008; De- vereux et al., 2010; Murphy et al., 2012; Pereira et al., 2013; Carlson et al., 2014) and semantic mod- els based on human behavioural estimation of con- ceptual features (Palatucci et al., 2009; Sudre et al., 2012; Chang et al., 2010; Bruffaerts et al., 2013; Fernandino et al., 2015) — have elucidated how dif- ferent brain regions contribute to semantic represen- tation of concrete nouns; however, how these results extend to non-concrete nouns is unknown. In computational modelling there has been in- creasing importance attributed to grounding seman- tic models in sensory modalities, e.g., Bruni et al. (2014), Kiela and Bottou (2014). Andrews et al. (2009) demonstrated that multi-modal models formed by combining text-based distributional in- formation with behaviourally generated conceptual properties (as a surrogate for perceptual experience) provide a better proxy for human-like intelligence. However, both the text-based and behaviourally- based components of their model were ultimately derived from linguistic information. Since then, in analyses of brain data, Anderson et al. (2013) have applied multi-modal models incorporating features that are truly grounded in natural image statistics to further support this claim. In addition, …

Content truncated to 500 words