Understanding language-elicited EEG data by predicting it from a fine-tuned language model

Proceedings of NAACL-HLT 2019, pages 43–57 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 43 Understanding language-elicited EEG data by predicting it from a fine-tuned language model Dan Schwartz Carnegie Mellon University 5000 Forbes Ave Pittsburgh, PA 15213 drschwar@cs.cmu.edu Tom Mitchell Carnegie Mellon University 5000 Forbes Ave Pittsburgh, PA 15213 tom.mitchell@cs.cmu.edu Abstract Electroencephalography (EEG) recordings of brain activity taken while participants read or listen to language are widely used within the cognitive neuroscience and psycholinguistics communities as a tool to study language com- prehension. Several time-locked stereotyped EEG responses to word-presentations – known collectively as event-related potentials (ERPs) – are thought to be markers for semantic or syntactic processes that take place during com- prehension. However, the characterization of each individual ERP in terms of what features of a stream of language trigger the response remains controversial. Improving this char- acterization would make ERPs a more use- ful tool for studying language comprehension. We take a step towards better understanding the ERPs by fine-tuning a language model to predict them. This new approach to analysis shows for the first time that all of the ERPs are predictable from embeddings of a stream of language. Prior work has only found two of the ERPs to be predictable. In addition to this analysis, we examine which ERPs bene- fit from sharing parameters during joint train- ing. We find that two pairs of ERPs previously identified in the literature as being related to each other benefit from joint training, while several other pairs of ERPs that benefit from joint training are suggestive of potential rela- tionships. Extensions of this analysis that fur- ther examine what kinds of information in the model embeddings relate to each ERP have the potential to elucidate the processes involved in human language comprehension. 1 Introduction The cognitive processes involved in human lan- guage comprehension are complex and only par- tially identified. According to the dual-stream model of speech comprehension (Hickok and Poeppel, 2007), sound waves are first converted to Figure 1: The electrodes from which each event-related potential was recorded in the data from Frank et al. (2015) (after figure 3 in (Frank et al., 2015)). The bot- tom portion of the figure shows a top-down schematic of the electrode locations with the nose facing towards the top of the page. Each ERP is the mean poten- tial from all of the indicated electrodes during a spe- cific time-window, creating a single scalar value per ERP per word. Overlapping circles indicate multiple ERPs recorded from the same electrode. The ELAN is measured from 125-175ms after stimulus onset, the LAN from 300-400ms, the N400 from 300ms-500ms, the EPNP from 400-600ms, the P600 from 500-700ms, and the PNP from 600-700ms. phoneme-like features and further processed by a ventral stream that maps those features onto words and semantic structures, and a dorsal stream that (among other things) supports audio-short term memory. The mapping of words onto meaning is thought to be subserved by widely distributed regions …

Content truncated to 500 words