Representations of language in a model of visually grounded speech signal

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 613–622 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1057 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 613–622 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1057 Representations of language in a model of visually grounded speech signal Grzegorz Chrupała Tilburg University g.chrupala@uvt.nl Lieke Gelderloos Tilburg University l.j.gelderloos@uvt.nl Afra Alishahi Tilburg University a.alishahi@uvt.nl Abstract We present a visually grounded model of speech perception which projects spoken utterances and images to a joint seman- tic space. We use a multi-layer recurrent highway network to model the temporal nature of spoken speech, and show that it learns to extract both form and meaning- based linguistic knowledge from the input signal. We carry out an in-depth analy- sis of the representations used by different components of the trained model and show that encoding of semantic aspects tends to become richer as we go up the hierar- chy of layers, whereas encoding of form- related aspects of the language input tends to initially increase and then plateau or de- crease. 1 Introduction Speech recognition is one of the success stories of language technology. It works remarkably well in a range of practical settings. However, this success relies on the use of very heavy supervi- sion where the machine is fed thousands of hours of painstakingly transcribed audio speech signal. Humans are able to learn to recognize and under- stand speech from notably weaker and noisier su- pervision: they manage to learn to extract struc- ture and meaning from speech by simply being ex- posed to utterances situated and grounded in their daily sensory experience. Modeling and emulat- ing this remarkable skill has been the goal of nu- merous studies; however in the overwhelming ma- jority of cases researchers used severely simplified settings where either the language input or the ex- tralinguistic sensory input, or both, are small scale and symbolically represented. Section 2 provides a brief overview of this research. More recently several lines of work have moved towards more realistic inputs while modeling or emulating language acquisition in a grounded set- ting. Gelderloos and Chrupała (2016) use the image captioning dataset MS COCO (Lin et al., 2014) to mimic the setting of grounded language learning: the sensory input consists of images of natural scenes, while the language input are pho- netically transcribed descriptions of these scenes. The use of such moderately large and low-level data allows the authors to train a multi-layer re- current neural network model, and to explore the nature and localization of the emerging hierarchy of linguistic representations learned in the process. Furthermore, in a series of recent studies Harwath and Glass (2015); Harwath et al. (2016); Harwath and Glass (2017) use image captioning datasets to model learning to understand spoken language from visual context with convolutional neural net- work models. Finally, there is a small but grow- ing body of …

Content truncated to 500 words