Combining text and vision in compound semantics: Towards a cognitively plausible multimodal model

Combining text and vision in compound semantics: Towards a cognitively plausible multimodal model Abhijeet Gupta† Fritz G¨unther∗ Ingo Plag† Laura Kallmeyer† Stefan Conrad† †Heinrich-Heine-Universit¨at D¨usseldorf {abhijeet.gupta,ingo.plag,kallmeyer,stefan.conrad}@uni-duesseldorf.de ∗University of T¨ubingen fritz.guenther@uni-tuebingen.de Abstract In the current state-of-the art distributional semantics model of the meaning of noun- noun compounds (such as chainsaw, but- terfly, home phone), CAOSS (Marelli et al. 2017), the semantic vectors of the individ- ual constituents are combined, and enriched by position-specific information for each con- stituent in its role as either modifier or head. Most recently there have been attempts to in- clude vision-based embeddings in these mod- els (G¨unther et al., 2020b), using the linear ar- chitecture implemented in the CAOSS model. In the present paper, we extend this line of research and demonstrate that moving to non- linear models improves the results for vision while linear models are a good choice for text. Simply concatenating text and vision vectors does not currently (yet) improve the prediction of human behavioral data over models using text- and vision-based measures separately. 1 Introduction The meaning and interpretation of noun-noun com- pounds, i.e. the combination of two words to form a new word (as in chainsaw, butterfly, home phone), is a contested area of study. In both theoretical linguistics and psycholinguistic circles one of the central questions is the contribution of the indi- vidual constituents in the construction of a com- pound’s meaning (see, e.g., Bauer et al. 2013; Bell and Sch¨afer 2016; Schmidtke et al. 2018, ch. 20 for recent discussion). Some psycho-computational approaches use dis- tributional semantic models to produce representa- tions of compound meanings. In the current state- of-the art model CAOSS (Marelli et al. 2017) the semantic vectors of the individual constituents are combined, and enriched by position-specific infor- mation for each constituent in its role as either modifier or head (e.g. chain as modifier in words like chainsaw, chain mail, chain reaction, chainsaw, chain-smoking)1. This enrichment is achieved by a linear architecture in which each constituent vector is first multiplied with a position-specific matrix before adding the two constituent representations to derive the compound representation. Another aspect of compound meaning has only recently begun to attract attention, namely the role of visual information in creating and processing in- dividual concepts and their combination. Research on embodied cognition revealed that concepts are not only based on linguistic experience, but are also grounded in perceptual experience (e.g. Barsalou 1999). In the field of neuro-psychological learning (e.g. Devereux et al. 2018), deep learning networks have been implemented in the learning of word meaning. Similarly, visual information should also play a major role in conceptual combination, at least for concrete concepts. The first study to show the effects of vision-based information in concep- tual combination has been (G¨unther et al., 2020b). In that study the authors compared two paral- lel implementations of the CAOSS model: one us- ing text-based embeddings (henceforth text embed- dings), the other picture-based semantic embed- dings (henceforth vision embeddings). These em- beddings (more specifically, the cosine …

Content truncated to 500 words