Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models

Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models Vesna G. Djokic† Jean Maillard‡ Luana Bulat‡ Ekaterina Shutova† †ILLC, University of Amsterdam, The Netherlands ‡Dept. of Computer Science & Technology, University of Cambridge, United Kingdom vesna@imsquared.eu, jean@maillard.it, ltf24@cam.ac.uk, e.shutova@uva.nl Abstract Recent years have seen a growing interest within the natural language processing (NLP) community in evaluating the ability of seman- tic models to capture human meaning represen- tation in the brain. Existing research has mainly focused on applying semantic models to de- code brain activity patterns associated with the meaning of individual words, and, more recently, this approach has been extended to sentences and larger text fragments. Our work is the first to investigate metaphor process- ing in the brain in this context. We evaluate a range of semantic models (word embed- dings, compositional, and visual models) in their ability to decode brain activity associated with reading of both literal and metaphoric sentences. Our results suggest that composi- tional models and word embeddings are able to capture differences in the processing of lit- eral and metaphoric sentences, providing sup- port for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension. 1 Introduction Distributionalsemanticsaims to representthe mean- ing of linguistic fragments as high-dimensional dense vectors. It has been successfully used to model the meaning of individual words in seman- tic similarity and analogy tasks (Mikolov et al., 2013; Pennington et al., 2014); as well as the meaning of larger linguistic units in a variety of tasks, such as translation (Bahdanau et al., 2014) and natural language inference (Bowman et al., 2015). Recent research has also demonstrated the ability of distributional models to predict patterns of brain activity associated with the meaning of words, obtained via functionalmagnetic resonance imaging (fMRI) (Mitchell et al., 2008; Devereux et al., 2010; Pereira et al., 2013). Following in their steps, Anderson et al. (2017b) have investigated visually grounded semantic models in this context. They found that while both visual and text-based models can equally decode concrete words, text- based models show an overall advantage over vi- sual models when decoding more abstract words. Other research has shown that data-driven semantic models can also successfully predict pat- terns of brain activity associated with the pro- cessing of sentences (Pereira et al., 2018) and larger narrative text passages (Wehbe et al., 2014; Huth et al., 2016). Recently, Jain and Huth (2018) investigated long short-term memory (LSTM) recurrent neural networks and showed that seman- tic models that incorporate larger-sized context windows outperform those with smaller-sized context windows, as well as the baseline bag- of-words model, in predicting brain activity associated with narrative listening. This suggests that compositional semantic models are suffi- ciently advanced to study the impact of linguistic context on semantic representation in the brain. In this paper, we investigate the extent to which lex- ical and compositional semantic models are able to capture differencesin human meaning represen- tations, resulting from meaning disambiguation of literal and metaphoric uses …

Content truncated to 500 words