From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1309–1319, Osaka, Japan, December 11-17 2016. From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning Lieke Gelderloos Tilburg University liekegelderloos@gmail.com Grzegorz Chrupała Tilburg University g.chrupala@uvt.nl Abstract We present a model of visually-grounded language learning based on stacked gated recurrent neural networks which learns to predict visual features given an image description in the form of a sequence of phonemes. The learning task resembles that faced by human language learners who need to discover both structure and meaning from noisy and ambiguous data across modalities. We show that our model indeed learns to predict features of the visual context given phonetically transcribed image descriptions, and show that it represents linguistic information in a hierarchy of levels: lower layers in the stack are comparatively more sensitive to form, whereas higher layers are more sensitive to meaning. 1 Introduction Children acquire their native language with little and weak supervision, exploiting noisy correlations between speech, visual, and other sensory signals, as well as via feedback from interaction with their peers and parents. Understanding this process is an important scientific challenge in its own right, but it also has potential to generate insights useful in engineering efforts to design conversational agents or robots. Computationally modeling the ability to learn linguistic form–meaning pairings has been the focus of much research, under scenarios simplified in a variety of ways, for example: • distributional learning from pure word-word co-occurrences with no perceptual grounding (Lan- dauer et al., 1998; Kiros et al., 2015); • cross-situational learning with word sequences and sets of symbols representing sensory input (Siskind, 1996; Fazly et al., 2010); • cross-situational learning using sensory audio and visual input, but with extremely limited sets of words and objects (Roy and Pentland, 2002; Iwahashi, 2003). Some recent models have used more naturalistic, larger-scale inputs, for example in cross-modal dis- tributional semantics (Lazaridou et al., 2015) or in implementations of the acquisition process trained on images paired with their descriptions (Chrupała et al., 2015). While in these works the representation of the visual scene consists of pixel-level perceptual data, the linguistic input consists of sentences seg- mented into discrete word symbols. In this paper we take a step towards addressing this major limitation, by using the phonetic transcription of input utterances. While this type of input is symbolic rather than perceptual, it goes a long way toward making the setting more naturalistic, and the acquisition prob- lem more challenging: the learner may need to discover structure corresponding to morphemes, words and phrases in an unsegmented string of phonemes, and the length of the dependencies that need to be detected grows substantially when compared to word-level models. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 1309 Our contributions We design and implement a simple architecture based on stacked recurrent neural networks with Gated Recurrent Units (Chung et al., 2014): our model …

Content truncated to 500 words