Interpretable Semantic Vectors from a Joint Model of Brain- and Text- Based Meaning

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 489–499, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Interpretable Semantic Vectors from a Joint Model of Brain- and Text- Based Meaning Alona Fyshe1, Partha P. Talukdar1, Brian Murphy2, Tom M. Mitchell1 1Machine Learning Department, Carnegie Mellon University 2School of Electronics, Electrical Engineering and Computer Science Queen’s University Belfast [afyshe,partha.talukdar,tom.mitchell]@cs.cmu.edu brian.murphy@qub.ac.uk Abstract Vector space models (VSMs) represent word meanings as points in a high dimen- sional space. VSMs are typically created using a large text corpora, and so repre- sent word semantics as observed in text. We present a new algorithm (JNNSE) that can incorporate a measure of semantics not previously used to create VSMs: brain activation data recorded while people read words. The resulting model takes advan- tage of the complementary strengths and weaknesses of corpus and brain activation data to give a more complete representa- tion of semantics. Evaluations show that the model 1) matches a behavioral mea- sure of semantics more closely, 2) can be used to predict corpus data for unseen words and 3) has predictive power that generalizes across brain imaging technolo- gies and across subjects. We believe that the model is thus a more faithful represen- tation of mental vocabularies. 1 Introduction Vector Space Models (VSMs) represent lexical meaning by assigning each word a point in high di- mensional space. Beyond their use in NLP appli- cations, they are of interest to cognitive scientists as an objective and data-driven method to discover word meanings (Landauer and Dumais, 1997). Typically, VSMs are created by collecting word usage statistics from large amounts of text data and applying some dimensionality reduction technique like Singular Value Decomposition (SVD). The basic assumption is that semantics drives a per- son’s language production behavior, and as a result co-occurrence patterns in written text indirectly encode word meaning. The raw co-occurrence statistics are unwieldy, but in the compressed VSM the distance between any two words is con- ceived to represent their mutual semantic similar- ity (Sahlgren, 2006; Turney and Pantel, 2010), as perceived and judged by speakers. This space then reflects the “semantic ground truth” of shared lex- ical meanings in a language community’s vocab- ulary. However corpus-based VSMs have been criticized as being noisy or incomplete representa- tions of meaning (Glenberg and Robertson, 2000). For example, multiple word senses collide in the same vector, and noise from mis-parsed sentences or spam documents can interfere with the final se- mantic representation. When a person is reading or writing, the se- mantic content of each word will be necessarily activated in the mind, and so in patterns of ac- tivity over individual neurons. In principle then, brain activity could replace corpus data as input to a VSM, and contemporary imaging techniques allow us to attempt this. Functional Magnetic Res- onance Imaging (fMRI) and Magnetoencephalog- raphy (MEG) are two brain activation recording technologies that measure neuronal activation in aggregate, and have been shown to have a pre- dictive …

Content truncated to 500 words