BrainBench: A Brain-Image Test Suite for Distributional Semantic Models

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2017–2021, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics BrainBench: A Brain-Image Test Suite for Distributional Semantic Models Haoyan Xu University of Victoria Victoria, BC, Canada exu@uvic.ca Brian Murphy Queen’s University Belfast Belfast, Northern Ireland, UK brian.murphy@qub.ac.uk Alona Fyshe University of Victoria Victoria, BC, Canada afyshe@uvic.ca∗ Abstract The brain is the locus of our language abil- ity, and so brain images can be used to ground linguistic theories. Here we introduce Brain- Bench, a lightweight system for testing dis- tributional models of word semantics. We compare the performance of several models, and show that the performance on brain-image tasks differs from the performance on behav- ioral tasks. We release our benchmark test as part of a web service. 1 Introduction There is active debate over how we should test se- mantic models. In fact, in 2016 there was an en- tire workshop dedicated to the testing of semantic representations (RepEval, 2016). Several before us have argued for the usage of brain data to test se- mantic models (Anderson et al., 2013; Murphy et al., 2012; Anderson et al., 2015), as a brain image repre- sents a snapshot of one person’s own semantic rep- resentation. Still, testing semantic models against brain imaging data is rarely done by those not in- timately involved in psycholinguistics or neurolin- guistics. This may be due to a lack of familiarity with neuroimaging methods and publicly available datasets. We present the first iteration of BrainBench, a new system that makes it easy to test seman- attic models using brain imaging data (Available http://www.langlearnlab.cs.uvic. ca/brainbench/). Our system has methodology that is similar to popular tests based on behavioral ∗Corresponding Author data (see Section 2.2), and has the additional benefit of being fast enough to offer as a web service. 2 The Tasks Here we outline the set of tasks we used to evaluate several popular Distributional Semantic (DS) mod- els. 2.1 Brain Image Data For BrainBench we use two brain image datasets collected while participants viewed 60 concrete nouns with line drawings (Mitchell et al., 2008; Su- dre et al., 2012). One dataset was collected us- ing fMRI (Functional Magnetic Resonance Imag- ing) and one with MEG (Magnetoencephalography). Each dataset has 9 participants, but the participants sets are disjoint, thus there are 18 unique partici- pants in all. Though the stimuli is shared across the two experiments, as we will see, MEG and fMRI are very different recording modalities and thus the data are not redundant. fMRI measures the change in blood oxygen levels in the brain, which varies according to the amount of work being done by a particular brain area. An fMRI image is a 3D volume of the brain where each point in the volume (called a voxel) represents brain activity at a particular place in the brain. In the fMRI dataset used here, each voxel represents a 3mm x 3mm x 5mm area of the brain. Each of …

Content truncated to 500 words