Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 291–301 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 291 Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech William N. Havard1,2, Laurent Besacier1, Jean-Pierre Chevrot2 1 LIG, Univ. Grenoble Alpes, CNRS, Grenoble INP, 38000 Grenoble, France 2 LIDILEM, Univ. Grenoble Alpes, 38000 Grenoble, France first-name.lastname@univ-grenoble-alpes.fr Abstract The language acquisition literature shows that children do not build their lexicon by seg- menting the spoken input into phonemes and then building up words from them, but rather adopt a top-down approach and start by seg- menting word-like units and then break them down into smaller units. This suggests that the ideal way of learning a language is by starting from full semantic units. In this paper, we in- vestigate if this is also the case for a neural model of Visually Grounded Speech trained on a speech-image retrieval task. We evalu- ated how well such a network is able to learn a reliable speech-to-image mapping when pro- vided with phone, syllable, or word boundary information. We present a simple way to in- troduce such information into an RNN-based model and investigate which type of boundary is the most efficient. We also explore at which level of the network’s architecture such infor- mation should be introduced so as to maximise its performances. Finally, we show that using multiple boundary types at once in a hierarchi- cal structure, by which low-level segments are used to recompose high-level segments, is ben- eficial and yields better results than using low- level or high-level segments in isolation. 1 Introduction and Prior Work Visually Grounded Speech (VGS) models whether CNN-based (Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017) or RNN-based (Chrupała et al., 2017; Merkx et al., 2019) became recently popular as they enable to model complex interaction between two modalities, namely speech and vision, and can thus be used to model child language acquisition, and more specifically lexical acquisition. Indeed, these models are trained to solve a speech-image retrieval task. That is, given a spoken input description, they are trained to re- trieve the image that matches the description the best. This task requires the model to identify lexi- cal units that might be relevant in the spoken input, detect which objects are present in the image, and fi- nally see if those objects match the detected spoken lexical units. Their task is thus very close to that of a child learning its mother tongue, who is sur- rounded by a visually perceptible context and tries to match parts of the acoustic input to surrounding visible situations. Research in language acquisition have put forward that children do not build their lex- icon by segmenting the spoken input into phonemes and then building up words, but rather adopt a top- down approach (Bortfeld et al., 2005) and start by identifying and memorising whole words (Jusczyk and Aslin, 1995) or chunks of words (Bannard and Matthews, 2008) and …

Content truncated to 500 words