Relating Simple Sentence Representations in Deep Neural Networks and the Brain

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5137–5154 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5137 Relating Simple Sentence Representations in Deep Neural Networks and the Brain Sharmistha Jat1∗ Hao Tang2 Partha Talukdar1 Tom Mitchell2 1Indian Institute of Science, Bangalore 2School of Computer Science, Carnegie Mellon University {sharmisthaj,ppt}@iisc.ac.in htang1@alumni.cmu.edu, tom.mitchell@cs.cmu.edu Abstract What is the relationship between sentence rep- resentations learned by deep recurrent mod- els against those encoded by the brain? there any correspondence between hidden lay-Is ers of these recurrent models and brain re- gions when processing sentences? Can these deep models be used to synthesize brain data which can then be utilized in other extrin- sic tasks? We investigate these questions us- ing sentences with simple syntax and seman- tics (e.g., The bone was eaten by the dog.). We consider multiple neural network archi- tectures, including recently proposed ELMo and BERT. We use magnetoencephalography (MEG) brain recording data collected from hu- man subjects when they were reading these simple sentences. Overall, we find that BERT’s activations cor- relate the best with MEG brain data. We also find that the deep network representation can be used to generate brain data from new sen- tences to augment existing brain data. To the best of our knowledge, this is the first work showing that the MEG brain recording when reading a word in a sentence can be used to distinguish earlier words in the sentence. Our exploration is also the first to use deep neural network representations to generate synthetic brain data and to show that it helps in improv- ing subsequent stimuli decoding task accuracy. 1 Introduction Deep learning methods for natural language pro- cessing have been very successful in a variety of Natural Language Processing (NLP) tasks. How- ever, the representation of language learned by such methods is still opaque. The human brain is an excellent language processing engine, and the brain representation of language is of course very effective. Even though both brain and deep ∗ This research was carried out during a research intern- ship at the Carnegie Mellon University. learning methods are representing language, the relationships among these representations are not thoroughly studied. Wehbe et al. (2014b) and Hale et al. (2018) studied this question in some limited capacity. Wehbe et al. (2014b) studied the pro- cessing of a story context at a word level during language model computation. Hale et al. (2018) studied the syntactic composition in RNNG model (Dyer et al., 2016) with human encephalography (EEG) data. We extend this line of research by investigating the following three questions: (1) what is the rela- tionship between sentence representations learned by deep learning networks and those encoded by the brain; (2) is there any correspondence between hidden layer activations in these deep models and brain regions; and (3) is it possible for deep re- current models to synthesize brain data so that they can effectively be used for brain data aug- mentation. In …

Content truncated to 500 words