Deconstructing multimodality: visual properties and visual context in human semantic processing

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 118–124 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 118 Deconstructing multimodality: visual properties and visual context in human semantic processing Christopher Davis†, Luana Bulat†, Anita Vero†, Ekaterina Shutova‡ † Department of Computer Science & Technology, University of Cambridge, U.K. {ccd38,luana.bulat,alv34}@cam.ac.uk ‡Institute for Logic, Language and Computation, University of Amsterdam, Netherlands e.shutova@uva.nl Abstract Multimodal semantic models that extend lin- guistic representations with additional percep- tual input have proved successful in a range of natural language processing (NLP) tasks. Recent research has successfully used neural methods to automatically create visual repre- sentations for words. However, these works have extracted visual features from complete images, and have not examined how differ- ent kinds of visual information impact perfor- mance. In contrast, we construct multimodal models that differentiate between internal vi- sual properties of the objects and their exter- nal visual context. We evaluate the models on the task of decoding brain activity associ- ated with the meanings of nouns, demonstrat- ing their advantage over those based on com- plete images. 1 Introduction Multimodal models combining linguistic and vi- sual information have enjoyed a growing interest in the field of semantics. Recent research has shown that such models outperform purely lin- guistic models on a range of NLP tasks, including modelling semantic similarity (Silberer and Lap- ata, 2014), lexical entailment (Kiela et al., 2015), and metaphor identification (Shutova et al., 2016). Despite this success, little is known about the na- ture of semantic information learned from images and why it is useful. For instance, some concepts may be better characterised by their own (inter- nal) visual properties and others by the (external) visual context, in which they appear. However, existing neural multimodal semantic approaches use entire images to learn visual word represen- tations, without differentiating between these two kinds of visual information. In contrast, we inves- tigate whether differentiating between internal vi- sual properties and external visual context is ben- eficial compared to learning visual representations from complete images. We construct three multi- modal models combining linguistic and visual in- formation: using (1) internal visual features ex- tracted from an object’s bounding box, (2) exter- nal visual features outside the bounding box, i.e. the visual context, and (3) visual features extracted from complete images. Figure 1 visualises the dif- ferent visual information extracted from an image. We use skip-gram (Mikolov et al., 2013) as our linguistic model and extract visual representations from a convolutional neural network (CNN) pre- trained on the ImageNet classification task (Fei- Fei, 2010). We evaluate the models in their ability to decode patterns of brain activity associated with the mean- ings of nouns, obtained via brain imaging. This choice of task allows us to assess the importance of each type of visual information in human se- mantic processing. Specifically, we perform two experiments: (1) using the Visual Genome (Kr- ishna et al., 2016) dataset of images where ob- jects are manually annotated with bounding boxes, …

Content truncated to 500 words