Encoding of phonology in a recurrent neural model of grounded speech

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 368–378, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Encoding of phonology in a recurrent neural model of grounded speech Afra Alishahi Tilburg University a.alishahi@uvt.nl Marie Barking Tilburg University m.barking@uvt.nl Grzegorz Chrupała Tilburg University g.chrupala@uvt.nl Abstract We study the representation and encod- ing of phonemes in a recurrent neural net- work model of grounded speech. We use a model which processes images and their spoken descriptions, and projects the visual and auditory representations into the same semantic space. We perform a number of analyses on how informa- tion about individual phonemes is encoded in the MFCC features extracted from the speech signal, and the activations of the layers of the model. Via experiments with phoneme decoding and phoneme discrim- ination we show that phoneme represen- tations are most salient in the lower lay- ers of the model, where low-level signals are processed at a fine-grained level, al- though a large amount of phonological information is retain at the top recurrent layer. We further find out that the at- tention mechanism following the top re- current layer significantly attenuates en- coding of phonology and makes the utter- ance embeddings much more invariant to synonymy. Moreover, a hierarchical clus- tering of phoneme representations learned by the network shows an organizational structure of phonemes similar to those pro- posed in linguistics. 1 Introduction Spoken language is a universal human means of communication. As such, its acquisition and rep- resentation in the brain is an essential topic in the study of the cognition of our species. In the field of neuroscience there has been a long-standing interest in the understanding of neural represen- tations of linguistic input in human brains, most commonly via the analysis of neuro-imaging data of participants exposed to simplified, highly con- trolled inputs. More recently, naturalistic data has been used and patterns in the brain have been cor- related with patterns in the input (e.g. Wehbe et al., 2014; Khalighinejad et al., 2017). This type of approach is relevant also when the goal is the understanding of the dynamics in com- plex neural network models of speech understand- ing. Firstly because similar techniques are of- ten applicable, but more importantly because the knowledge of how the workings of artificial and biological neural networks are similar or different is valuable for the general enterprise of cognitive science. Recent studies have implemented models which learn to understand speech in a weakly and in- directly supervised fashion from correlated audio and visual signal: Harwath et al. (2016); Har- wath and Glass (2017); Chrupała et al. (2017a). This is a departure from typical Automatic Speech Recognition (ASR) systems which rely on large amounts of transcribed speech, and these recent models come closer to the way humans acquire language in a grounded setting. It is thus es- pecially interesting to investigate to what extent the traditional levels of linguistic analysis such as phonology, morphology, syntax and semantics …

Content truncated to 500 words