Multi- and Cross-Modal Semantics Beyond Vision: Grounding in Auditory Perception

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2461–2470, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Multi- and Cross-Modal Semantics Beyond Vision: Grounding in Auditory Perception Douwe Kiela Computer Laboratory University of Cambridge douwe.kiela@cl.cam.ac.uk Stephen Clark Computer Laboratory University of Cambridge stephen.clark@cl.cam.ac.uk Abstract Multi-modal semantics has relied on fea- ture norms or raw image data for per- ceptual input. In this paper we examine grounding semantic representations in raw auditory data, using standard evaluations for multi-modal semantics, including mea- suring conceptual similarity and related- ness. We also evaluate cross-modal map- pings, through a zero-shot learning task mapping between linguistic and auditory modalities. In addition, we evaluate multi- modal representations on an unsupervised musical instrument clustering task. To our knowledge, this is the first work to com- bine linguistic and auditory information into multi-modal representations. 1 Introduction Although distributional models (Turney and Pan- tel, 2010; Clark, 2015) have proved useful for a variety of NLP tasks, the fact that the meaning of a word is represented as a distribution over other words implies that they suffer from the ground- ing problem (Harnad, 1990); i.e. they do not ac- count for the fact that human semantic knowledge is grounded in the perceptual system (Louwerse, 2008). Motivated by human concept acquisition, multi-modal semantics enhances linguistic repre- sentations with extra-linguistic perceptual input. These models outperform language-only models on a range of tasks, including modelling semantic similarity and relatedness, and predicting compo- sitionality (Silberer and Lapata, 2012; Roller and Schulte im Walde, 2013; Bruni et al., 2014). Al- though feature norms have also been used, raw image data has become the de-facto perceptual modality in multi-modal models. However, if the objective is to ground seman- tic representations in perceptual information, why stop at image data? The meaning of violin is surely not only grounded in its visual properties, such as shape, color and texture, but also in its sound, pitch and timbre. To understand how per- ceptual input leads to conceptual representation, we should use as many perceptual modalities as possible. A recent preliminary study by Lopopolo and van Miltenburg (2015) found that it is possible to derive uni-modal semantic representations from sound data. Here, we explore taking multi-modal semantics beyond its current reliance on image data and experiment with grounding semantic rep- resentations in the auditory perceptual modality. Multi-modal models that rely on raw image data have typically used “bag of visual words” (BoVW) representations (Sivic and Zisserman, 2003). We follow a similar approach for the auditory modality and construct bag of audio words (BoAW) representations. Following pre- vious work in multi-modal semantics, we evalu- ate these models on measuring conceptual simi- larity and relatedness, and inducing cross-modal mappings between modalities to perform zero- shot learning. In addition, we evaluate on an unsupervised musical instrument clustering task. Our findings indicate that multi-modal representa- tions enriched with auditory information perform well on relatedness and similarity tasks, particu- larly on words that have auditory assocations. To our knowledge, …

Content truncated to 500 words