Quantitative modeling of the neural representation of adjective-noun phrases to account for fMRI activation

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 638–646, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Quantitative modeling of the neural representation of adjective-noun phrases to account for fMRI activation Kai-min K. Chang1 Vladimir L. Cherkassky2 Tom M. Mitchell3 Marcel Adam Just2 Language Technologies Institute1 Center for Cognitive Brain Imaging2 Machine Learning Department3 Carnegie Mellon University Pittsburgh, PA 15213, U.S.A. {kkchang,cherkassky,tom.mitchell,just}@cmu.edu Abstract Recent advances in functional Magnetic Resonance Imaging (fMRI) offer a significant new approach to studying semantic represen- tations in humans by making it possible to di- rectly observe brain activity while people comprehend words and sentences. In this study, we investigate how humans compre- hend adjective-noun phrases (e.g. strong dog) while their neural activity is recorded. Classi- fication analysis shows that the distributed pattern of neural activity contains sufficient signal to decode differences among phrases. Furthermore, vector-based semantic models can explain a significant portion of system- atic variance in the observed neural activity. Multiplicative composition models of the two-word phrase outperform additive models, consistent with the assumption that people use adjectives to modify the meaning of the noun, rather than conjoining the meaning of the adjective and noun. 1 Introduction How humans represent meanings of individual words and how lexical semantic knowledge is combined to form complex concepts are issues fundamental to the study of human knowledge. There have been a variety of approaches from different scientific communities trying to charac- terize semantic representations. Linguists have tried to characterize the meaning of a word with feature-based approaches, such as semantic roles (Kipper et al., 2006), as well as word-relation approaches, such as WordNet (Miller, 1995). Computational linguists have demonstrated that a word’s meaning is captured to some extent by the distribution of words and phrases with which it commonly co-occurs (Church & Hanks, 1990). Psychologists have studied word meaning through feature-norming studies (Cree & McRae, 2003) in which human participants are asked to list the features they associate with various words. There are also efforts to recover the latent semantic structure from text corpora using tech- niques such as LSA (Landauer & Dumais, 1997) and topic models (Blei et al., 2003). Recent advances in functional Magnetic Resonance Imaging (fMRI) provide a significant new approach to studying semantic representations in humans by making it possible to directly observe brain activity while people comprehend words and sentences. fMRI measures the hemodynamic response (changes in blood flow and blood oxygenation) related to neural activity in the human brain. Images can be acquired at good spatial resolution and reason- able temporal resolution – the activity level of 15,000 - 20,000 brain volume elements (voxels) of about 50 mm3 each can be measured every 1 second. Recent multivariate analyses of fMRI activity have shown that classifiers can be trained to decode which of several visually pre- sented objects or object categories a person is contemplating, given the person’s fMRI- measured neural activity (Cox and Savoy, 2003; O'Toole et al., 2005; Haynes and …

Content truncated to 500 words