Modeling Affirmative and Negated Action Processing in the Brain with Lexical and Compositional Semantic Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5155–5165 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5155 Modeling affirmative and negated action processing in the brain with lexical and compositional semantic models Vesna G. Djokic♠ Jean Maillard♣ Luana Bulat♣ Ekaterina Shutova♦ ♠Department of Neuroscience, University of Southern California, USA ♣Dept. of Computer Science & Technology, University of Cambridge, United Kingdom ♦ILLC, University of Amsterdam, The Netherlands vesna@imsquared.eu, jean@maillard.it, ltf24@cam.ac.uk, e.shutova@uva.nl Abstract Recent work shows that distributional seman- tic models can be used to decode patterns of brain activity associated with individual words and sentence meanings. However, it is yet un- clear to what extent such models can be used to study and decode fMRI patterns associated with specific aspects of semantic composition such as the negation function. In this paper, we apply lexical and compositional seman- tic models to decode fMRI patterns associated with negated and affirmative sentences con- taining hand-action verbs. Our results show reduced decoding (correlation) of sentences where the verb is in the negated context, as compared to the affirmative one, within brain regions implicated in action-semantic process- ing. This supports behavioral and brain imag- ing studies, suggesting that negation involves reduced access to aspects of the affirmative mental representation. The results pave the way for testing alternate semantic models of negation against human semantic processing in the brain. 1 Introduction Computational semantic models are increasingly being evaluated in their ability to capture aspects of human semantic processing, including similar- ity and association judgments (De Deyne et al., 2016) and semantic representation in the brain (Bulat et al., 2017). Prior work shows that dis- tributional semantic models (DSMs) are able to decode functional magnetic resonance imaging (fMRI) patterns associated with the meaning of concrete words (Anderson et al., 2013). Relevant to our work, Carota et al. (2017) showed that the similarity structure of DSMs for action words cor- relates with that of fMRI patterns in brain regions implicated in action-semantic processing. More recent studies have also investigated the ability of DSMs to predict fMRI patterns of sen- tential meanings (Pereira et al., 2018) and larger narrative text passages (Wehbe et al., 2014; Huth et al., 2016). They have shown that encoding mod- els based on word embeddings are able to capture subtle aspects of sentence meaning in the brain, even when these models are oblivious of word or- der and syntactic structure. While promising, none of this research has so far systematically investi- gated specific semantic composition phenomena and processing at the syntax-semantic interface, such as that of the negation function. Negation is a fundamental abstraction necessary for efficient reasoning and communication (Horn, 1989). Although it is typically marked syntac- tically, the semantics of negation in natural lan- guage usage has proven to be rather challenging to pinpoint (Speranza and Horn, 2010). In logi- cal negation, the negation operator has been suc- cinctly described as a truth-functional operation, reversing the truth value of a sentence. On …

Content truncated to 500 words