Multimodal Semantic Learning from Child-Directed Input

Proceedings of NAACL-HLT 2016, pages 387–392, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Multimodal Semantic Learning from Child-Directed Input Angeliki Lazaridou University of Trento angeliki.lazaridou@unitn.it Grzegorz Chrupała Tilburg University g.chrupala@uvt.nl Raquel Fern´andez University of Amsterdam raquel.fernandez@uva.nl Marco Baroni University of Trento marco.baroni@unitn.it Abstract Children learn the meaning of words by being exposed to perceptually rich situations (lin- guistic discourse, visual scenes, etc). Current computational learning models typically sim- ulate these rich situations through impover- ished symbolic approximations. In this work, we present a distributed word learning model that operates on child-directed speech paired with realistic visual scenes. The model inte- grates linguistic and extra-linguistic informa- tion (visual and social cues), handles referen- tial uncertainty, and correctly learns to asso- ciate words with objects, even in cases of lim- ited linguistic exposure. 1 Introduction Computational models of word learning typically approximate the perceptual context that learners are exposed to through artificial proxies, e.g., represent- ing a visual scene via a collection of symbols such as cat and dog, signaling the presence of a cat, a dog, etc. (Yu and Ballard, 2007; Fazly et al., 2010, inter alia).1 While large amounts of data can be generated in this way, they will not display the complexity and richness of the signal found in the natural environment a child is exposed to. We take a step towards a more realistic setup by intro- ducing a model that operates on naturalistic images of the objects present in a communicative episode. Inspired by recent computational models of mean- ing (Bruni et al., 2014; Kiros et al., 2014; Silberer 1See K´ad´ar et al. (2015) for a recent review of this line of work, and another learning model using, like ours, real visual input. and Lapata, 2014), that integrate distributed linguis- tic and visual information, we build upon the Multi- modal Skip-Gram (MSG) model of Lazaridou et al. (2015). and enhance it to handle cross-referential uncertainty. Moreover, we extend the cues com- monly used in multimodal learning (e.g., objects in the environment) to include social cues (e.g., eye- gaze, gestures, body posture, etc.) that reflect speak- ers’ intentions and generally contribute to the un- folding of the communicative situation (Stivers and Sidnell, 2005). As a first step towards developing full-fleged learning systems that leverage all signals available within a communicative setup, in our ex- tended model we incorporate information regarding the objects that caregivers are holding. 2 Attentive Social MSG Model Like the original MSG, our model learns multimodal word embeddings by reading an utterance sequen- tially and making, for each word, two sets of pre- dictions: (a) the preceding and following words, and (b) the visual representations of objects co-occurring with the utterance. However, unlike Lazaridou et al. (2015), we do not assume we know the right object to be associated with a word. We consider instead a more realistic scenario where multiple words in an utterance co-occur with multiple objects in the corresponding scene. Under this referential uncer- tainty, the model needs …

Content truncated to 500 words