Linking artificial and human neural representations of language

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 529–539, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 529 Linking artificial and human neural representations of language Jon Gauthier and Roger P. Levy Massachusetts Institute of Technology Department of Brain and Cognitive Sciences jon@gauthiers.net, rplevy@mit.edu Abstract What information from an act of sentence un- derstanding is robustly represented in the hu- man brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be pre- dicted from the fMRI signal evoked by the sen- tence. We take a pre-trained BERT architec- ture as a baseline sentence encoding model and fine-tune it on a variety of natural language un- derstanding (NLU) tasks, asking which lead to improvements in brain-decoding performance. We find that none of the sentence encoding tasks tested yield significant increases in brain decoding performance. Through further task ablations and representational analyses, we find that tasks which produce syntax-light rep- resentations yield significant improvements in brain decoding performance. Our results con- strain the space of NLU models that could best account for human neural representations of language, but also suggest limits on the possi- bility of decoding fine-grained syntactic infor- mation from fMRI human neuroimaging. What are the neural representations which sup- port human language understanding? Language neuroscience has converged on a set of reliable physiological markers related to language process- ing (Kutas and Federmeier, 2011), and a picture of where in the brain different aspects of language pro- cessing take place (Fedorenko et al., 2010). But we still largely lack a coherent picture of the structure and format of the neural representations driving language understanding. Part of this struggle is due to a scarcity of can- didate representational structures in the first place. While there are certainly enough representational Source code for all analyses reported in this paper is available at http://bit.ly/nn-decoding. theories of language understanding, many are spec- ified at too high a level of analysis to plausibly map onto neural structures without serious further revision (Poeppel, 2012). Studies which draw on these high-level repre- sentations must therefore also assume some link between such representations and measures of neu- ral activity — for example, that the magnitude of neural activations should match up with the proba- bility values derived from a computational model (Brennan et al., 2016), or measures derived from syntactic representations of the input (Pallier et al., 2011). While the success of these mapping studies demonstrates that specific summary statistics of lin- guistic representations have correlates in the mind, these summary statistics are not themselves candi- date representations for the fundamental operations underlying language understanding. In the meantime, research in natural language processing has produced neural network models that capture many different sorts of intelligent lan- guage understanding behavior (Collobert et al., 2011; Goldberg, 2016). These models accomplish this behavior in an implementation better …

Content truncated to 500 words