Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 339–348 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 339 Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech William N. Havard1,2, Jean-Pierre Chevrot2, Laurent Besacier1 1 LIG, Univ. Grenoble Alpes, CNRS, Grenoble INP, 38000 Grenoble, France 2 LIDILEM, Univ. Grenoble Alpes, 38000 Grenoble, France first-name.lastname@univ-grenoble-alpes.fr Abstract In this paper, we study how word-like units are represented and activated in a recurrent neu- ral model of visually grounded speech. The model used in our experiments is trained to project an image and its spoken description in a common representation space. We show that a recurrent model trained on spoken sentences implicitly segments its input into word-like units and reliably maps them to their correct visual referents. We introduce a methodology originating from linguistics to analyse the rep- resentation learned by neural networks – the gating paradigm – and show that the correct representation of a word is only activated if the network has access to first phoneme of the tar- get word, suggesting that the network does not rely on a global acoustic pattern. Furthermore, we find out that not all speech frames (MFCC vectors in our case) play an equal role in the final encoded representation of a given word, but that some frames have a crucial effect on it. Finally, we suggest that word representation could be activated through a process of lexical competition. 1 Introduction Neural models of Visually Grounded Speech (VGS) sparked interest in linguists and cognitive scientists as they are able to incorporate multi- ple modalities in a single network and allow the analysis of complex interactions between them. Analysing these models does not only help to un- derstand their technological limitations, but may also yield insight on the cognitive processes at work in humans (Dupoux, 2018) who learn from contextually grounded speech utterances (either visually, haptically, socially, etc.). This is with this idea in mind that one of the first computa- tional model of visually grounded word acquisi- tion was introduced by Roy and Pentland (2002). More recently, Harwath et al. (2016) and Chrupała et al. (2017) were among the first to propose neural models integrating these two modalities. While Chrupała et al. (2017) and Alishahi et al. (2017) focused on analysing speech representa- tions learnt by speech-image neural models from a phonological and semantic point of view, the present work focuses on lexical acquisition and the way speech utterances are segmented into lexical units by a neural model. More precisely, we aim at understanding how word-like units are processed by a VGS architec- ture. First, we study if such models are robust to isolated word stimuli. As such networks are trained on raw speech utterances, robustness to isolated word stimuli would indicate that a seg- mentation process was implicitly carried out at training time. We also explore which factors in- fluence the most such word recognition. In a second step, to better understand how individual words …

Content truncated to 500 words