Exploring Semantic Representation in Brain Activity Using Word Embeddings

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 669–679, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Exploring Semantic Representation in Brain Activity Using Word Embeddings Yu-Ping Ruan1, Zhen-Hua Ling1 and Yu Hu1,2 1National Engineering Laboratory for Speech and Language Information Processing University of Science and Technology of China, Hefei, China 2iFLYTEK Research, Hefei, China ypruan@mail.ustc.edu.cn, zhling@ustc.edu.cn, yuhu@iflytek.com Abstract In this paper, we utilize distributed word rep- resentations (i.e., word embeddings) to anal- yse the representation of semantics in brain activity. The brain activity data were recorded using functional magnetic resonance imaging (fMRI) when subjects were viewing words. First, we analysed the functional selectivity of different cortex areas by calculating the corre- lations between neural responses and several types of word representations, including skip- gram word embeddings, visual semantic vec- tors, and primary visual features. The results demonstrated consistency with existing neu- roscientific knowledge. Second, we utilized behavioural data as the semantic ground truth to measure their relevance with brain activity. A method to estimate word embeddings under the constraints of brain activity similarities is further proposed based on the semantic word embedding (SWE) model. The experimental results show that the brain activity data are sig- nificantly correlated with the behavioural data of human judgements on semantic similarity. The correlations between the estimated word embeddings and the semantic ground truth can be effectively improved after integrating the brain activity data for learning, which implies that semantic patterns in neural rep- resentations may exist that have not been fully captured by state-of-the-art word embeddings derived from text corpora. 1 Introduction Recently, the topic of exploring semantic represen- tation in human brain has attracted the attention of researchers from both neuroscience and computa- tional linguistics fields. In these studies, concepts are represented in terms of neural activation patterns in the brain that can be recorded by functional magnetic resonance imaging (fMRI) (Haxby et al., 2001). It has been found that the semantic space shared among different individuals is distributed continuously across the cortex (Huth et al., 2012). A recent study proposed an efficient way to measure and visualize the semantic selectivity of different cortex areas (Huth et al., 2016). Similar to the distributed semantic representation in the brain, describing the meaning of a word using a dense low-dimensional and continuous vec- tor (i.e., word embedding) is currently a popular approach in computational linguistics (Hinton et al., 1986; Turney et al., 2010). Word embeddings are commonly estimated from large text corpora utilizing statistics concerning the co-occurrences of words (Mikolov et al., 2013a; Mikolov et al., 2013b; Pennington et al., 2014). To investigate the correlation between word embeddings and the brain activity involved in viewing words, Mitchell et al. (2008) designed a computational model to predict brain responses using hand-tailored word embeddings as input. Further, Fyshe et al. (2014) proposed a joint non-negative sparse embedding (JNNSE) method to combine fMRI data and textual data to estimate word embeddings. This work improved the correlation between word embeddings and …

Content truncated to 500 words