Sound-Word2Vec: Learning Word Representations Grounded in Sounds

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 920–925 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Sound-Word2Vec: Learning Word Representations Grounded in Sounds Ashwin Vijayakumar1, Ramakrishna Vedantam2 and Devi Parikh1 1 Georgia Tech, 2 Virgina Tech {ashwinkv,parikh}@gatech.edu, vrama1@vt.edu Abstract To be able to interact better with humans, it is crucial for machines to understand sound – a primary modality of human per- ception. Previous works have used sound to learn embeddings for improved generic semantic similarity assessment. In this work, we treat sound as a first-class cit- izen, studying downstream 6textual tasks which require aural grounding. To this end, we propose sound-word2vec – a new embedding scheme that learns specialized word embeddings grounded in sounds. For example, we learn that two seemingly (se- mantically) unrelated concepts, like leaves and paper are similar due to the similar rustling sounds they make. Our embed- dings prove useful in textual tasks requir- ing aural reasoning like text-based sound retrieval and discovering Foley sound ef- fects (used in movies). Moreover, our em- bedding space captures interesting depen- dencies between words and onomatopoeia and outperforms prior work on aurally- relevant word relatedness datasets such as AMEN and ASLex. 1 Introduction Sound and vision are the dominant perceptual sig- nals, while language helps us communicate com- plex experiences via rich abstractions. For exam- ple, a novel can stimulate us to mentally construct the image of the scene despite having never phys- ically perceived it. Indeed, language has evolved to contain numerous constructs that help depict vi- sual concepts. For example, we can easily form the picture of a white, furry cat with blue eyes via. a description of the cat in terms of its visual at- tributes (Lampert et al., 2009; Parikh and Grau- man, 2011). Need for Onomatopoeia. However, how would one describe the auditory instantiation of cats? While a first thought might be to use audio de- scriptors like loud, shrill, husky etc. as mid-level constructs or “attributes”, arguably, it is difficult to precisely convey and comprehend sound through such language. Indeed, Wake and Asahi (1998) find that humans first communicate sounds using “onomatopoeia” – words that are suggestive of the phonetics of sounds while having no explicit meaning e.g. meow, tic-toc. When asked for further explanation of sounds, humans provide descriptions of potential sound sources or impres- sions created by the sound (pleasant, annoying, etc.) Need for Grounding in Sound. While ono- matopoeic words exist for commonly found concepts, a vast majority of concepts are not as perceptually striking or sufficiently frequent for us to come up with dedicated words describing their sounds. Even worse, some sounds, say, musical instruments, might be difficult to mimic using speech. Thus, for a large number of concepts there seems to be a gap between sound and its counterpart in language (Sundaram and Narayanan, 2006). This becomes problematic in specific situations where we want to talk about the heavy tail of concepts and their sounds, or while describing a …

Content truncated to 500 words