Learning English with Peppa Pig

Learning English with Peppa Pig Mitja Nikolaus Aix-Marseille University, France mitja.nikolaus@univ-amu.fr Afra Alishahi Tilburg University, The Netherlands a.alishahi@uvt.nl Grzegorz Chrupała Tilburg University, The Netherlands grzegorz@chrupala.me Abstract Recent computational models of the acqui- sition of spoken language via grounding in perception exploit associations between spo- ken and visual modalities and learn to represent speech and visual data in a joint vector space. A major unresolved issue from the point of eco- logical validity is the training data, typically consisting of images or videos paired with spo- ken descriptions of what is depicted. Such a setup guarantees an unrealistically strong cor- relation between speech and the visual data. In the real world the coupling between the linguistic and the visual modality is loose, and often confounded by correlations with non-semantic aspects of the speech signal. Here we address this shortcoming by using a dataset based on the children’s cartoon Peppa Pig. We train a simple bi-modal architecture on the portion of the data consisting of dialog between characters, and evaluate on segments containing descriptive narrations. Despite the weak and confounded signal in this training data, our model succeeds at learning aspects of the visual semantics of spoken language. 1 Introduction Attempts to model or simulate the acquisition of spoken language via grounding in the visual modality date to the beginning of this century (Roy and Pentland, 2002) but have gained momen- tum recently with the revival of neural networks (e.g., Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Chrupała et al., 2017; Alishahi et al., 2017; Harwath et al., 2018; Merkx et al., 2019; Havard et al., 2019a; Rouditchenko et al., 2021; Khorrami and R¨as¨anen, 2021; Peng and Harwath, 2022). Current approaches work well enough from an applied point of view, but most are not generalizable to real-life situations that humans or adaptive artificial agents expe- rience. Commonly used training data consist of images or videos paired with spoken descriptions of the scene depicted: However, the type of input that a language learner receives from the environ- ment is much more challenging. Firstly, speech is only loosely coupled with the visual modality in naturalistic settings (Matusevych et al., 2013; Beekhuizen et al., 2013). Speakers often mention concepts that are not present in the immediate perceptual context, or talk about events that are remote in space and/or time (for example past experiences or future plans). Secondly, in addition to correlations between the visual scenes and the meaning of spo- ken utterances, there are also correlations with non-semantic aspects of the speech signal, such as the voices of specific speakers, as well as with non-speech ambient sounds. Although it is plausible that such non-semantic correlations can sometimes be useful to the learner in the general endeavor of making sense of the world, for the specific task of learning the semantics of linguistic units they are likely more often an obstacle, as they make it harder to zoom in on the meaning-bearing aspects of the audio signal. In the current study …

Content truncated to 500 words