Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication

Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication Gaˇsper Beguˇs University of California, Berkeley, USA begus@berkeley.edu Abstract This paper models unsupervised learning of an identity-based pattern (or copying) in speech called reduplication from raw continuous data with deep convolutional neural networks. We use the ciwGAN architecture (Beguˇs, 2021a) in which learning of meaningful representa- tions in speech emerges from a requirement that the CNNs generate informative data. We propose a technique to wug-test CNNs trained on speech and, based on four generative tests, argue that the network learns to represent an identity-based pattern in its latent space. By manipulating only two categorical variables in the latent space, we can actively turn an unreduplicated form into a reduplicated form with no other substantial changes to the output in the majority of cases. We also argue that the network extends the identity-based pattern to unobserved data. Exploration of how meaning- ful representations of identity-based patterns emerge in CNNs and how the latent space variables outside of the training range corre- late with identity-based patterns in the output has general implications for neural network interpretability. 1 Introduction The relationship between symbolic representa- tions and connectionism has been subject to ongoing discussions in computational cognitive science. Phonology offers a unique testing ground in this debate as it is concerned with the first discretization that human language users per- form: from continuous phonetic data to discretized mental representations of meaning-distinguishing sounds called phonemes. Identity-based patterns, repetition, or copying have been at the center of this debate (Marcus et al., 1999). Reduplication is a morphophonolo- gical process where phonological content (pho- nemes) get copied from a word (also called the base) with some added meaning (Inkelas and Zoll, 2005; Urbanczyk, 2017). It can be total, which means that all phonemes in a word get copied (e.g., /pula/ → [pula-pula]), or partial, where only a subset of segments gets copied (e.g., /pula/ → [pu-pula]). Reduplication is indeed unique among pro- cesses in natural language because combining learned entities based on training data distributions does not yield the desired outputs. For example, a learner can be presented with pairs of bare and reduplicated words, such as /pala/ ∼ /papala/ and /tala/ ∼ /tatala/. The learner can then be tested on providing a reduplicated variant of a novel unob- served item with an initial sound /k/ that they have not been exposed to (e.g., /kala/). If the learner learns the reduplication pattern, they will output [kakala]. If the learner simply learns that /pa/ and /ta/ are optional constituents that can be attached to words based on data distribution, they will out- put [pakala] or [takala]. Reduplication is thus an identity-based pattern (similar to copying), which is computationally more challenging to learn (Gasser, 1993), both in connectionist (Brugiapaglia et al., 2020) and non-connectionist frameworks (Savitch, 1989; Dolatian and Heinz, 2020). In /kiajkiajla/, the two sounds in the reduplicative morpheme, /ki/ and /aj/, need to be in an identity relationship with the first two segments of the base, …

Content truncated to 500 words