Speaking, Seeing, Understanding: Correlating semantic models with conceptual representation in the brain

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1081–1091 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Speaking, Seeing, Understanding: Correlating semantic models with conceptual representation in the brain Luana Bulat, Stephen Clark and Ekaterina Shutova Computer Laboratory University of Cambridge {luana.bulat,stephen.clark,ekaterina.shutova}@cl.cam.ac.uk Abstract Research in computational semantics is in- creasingly guided by our understanding of human semantic processing. However, semantic models are typically studied in the context of natural language process- ing system performance. In this paper, we present a systematic evaluation and com- parison of a range of widely-used, state- of-the-art semantic models in their abil- ity to predict patterns of conceptual repre- sentation in the human brain. Our results provide new insights both for the design of computational semantic models and for further research in cognitive neuroscience. 1 Introduction Recent years have witnessed many breakthroughs in data-driven semantic modelling: from the log- linear skip-gram model of Mikolov et al. (2013a) to multi-modal meaning representations (Bruni et al., 2012; Kiela and Bottou, 2014; Kiela and Clark, 2015; Kiela et al., 2015a). These models boast of a higher performance accuracy in numer- ous semantic tasks, including modeling seman- tic similarity and relatedness (Silberer and Lap- ata, 2012), lexical entailment (Kiela et al., 2015b), analogy (Mikolov et al., 2013b) and metaphor (Shutova et al., 2016). However, less is known about the extent to which such models correlate with and reflect human conceptual representation. Much research in the cognitive neuroscience community has been concerned with uncovering how the brain represents conceptual knowledge, by leveraging brain activation data associated with the meanings of concepts obtained during func- tional magnetic resonance imaging (fMRI) exper- iments. In the computational linguistics commu- nity, the availability of such fMRI data provides researchers with a benchmark for evaluating se- mantic model performance in terms of their abil- ity to represent human semantic memory. Mitchell et al. (2008) were the first to demonstrate that distributional semantic models encode some of the patterns found in the fMRI data. Other re- searchers followed in their steps, evaluating tra- ditional count-based distributional models (Dev- ereux et al., 2010; Murphy et al., 2012), topic model-based semantic features (Pereira et al., 2013), psycholinguistic and behavioural features (Palatucci et al., 2009; Chang et al., 2010; Fer- nandino et al., 2015) and visual representations (Anderson et al., 2013, 2017). While all of these studies report correlation between the investigated semantic models and patterns found in the brain imaging data, their focus on individual models and the use of different datasets and prediction meth- ods make their results difficult to compare and to integrate into a coherent evaluation landscape. The work of Murphy et al. (2012) is an exception, in that the authors systematically compare several distributional models with a range of parameters on the same brain imaging dataset. However, they focus on the traditional count-based distributional models only. We take inspiration from the works of Mitchell et al. (2008) and Murphy et al. (2012); however, we conduct a more …

Content truncated to 500 words