Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3220 - 3237 July 10-15, 2022 ©2022 Association for Computational Linguistics Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity? Subba Reddy Oota1,2*, Jashn Arora2*, Veeral Agarwal2, Mounika Marreddy2 Manish Gupta2,3 and Bapi Raju Surampudi2 1INRIA, Bordeaux, France; 2IIIT Hyderabad, India; 3Microsoft, India subba-reddy.oota@inria.fr, jashn.arora@research.iiit.ac.in veeral.agarwal@research.iiit.ac.in, mounika.marreddy@research.iiit.ac.in gmanish@microsoft.com, raju.bapi@iiit.ac.in Abstract Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Trans- former models and has not explored the effi- cacy of task-specific learned Transformer rep- resentations. In this work, we explore trans- fer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predict- ing brain responses from two diverse datasets: Pereira (subjects reading sentences from para- graphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in dif- ferent regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase gen- eration, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predic- tive brain activity versus language right hemi- sphere, (ii) posterior medial cortex, temporo- parieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp. 1 Introduction Brain encoding aims at constructing neural brain activity given an input stimulus. Since the discov- ery of the relationship between language stimuli and functions of brain networks using fMRI [for ex., (Constable et al., 2004)], researchers have been interested in understanding how the neural encod- ing models predict the fMRI brain activity. Sev- eral brain encoding models have been developed to (i) understand the ventral stream in biological vision (Yamins et al., 2014; Kietzmann et al., 2019; Bao et al., 2020), and (ii) to study the higher-level cognition like language processing (Gauthier and Levy, 2019; Schrimpf et al., 2021; Schwartz et al., 2019). Some recent studies (Nishida et al., 2015; Huth et al., 2016) have been able to identify brain ROIs (Region of Interest) that respond to words that have a similar meaning and have thus built a “semantic atlas” of how the human brain organizes language. Further, several studies (Oota et al., 2018; Jain and Huth, 2018; Hollenstein et al., 2019) have used a wide variety of word embeddings where words represented as vectors in an embedding space are mapped to brain activation for improved neural coding. Recently, Transformer (Vaswani et al., 2017) based models like BERT (Devlin et al., 2019) have been found to be very effective across a large num- ber …

Content truncated to 500 words