Large-Scale Representation Learning from Visually Grounded Untranscribed Speech

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 55–65 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 55 Large-scale representation learning from visually grounded untranscribed speech Gabriel Ilharco†∗ Yuan Zhang‡ Jason Baldridge‡ †Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA ‡Google Research, Mountain View, CA, USA gamaga@cs.washington.edu, {zhangyua,jridge}@google.com Abstract Systems that can associate images with their spoken audio captions are an important step towards visually grounded language learning. We describe a scalable method to automati- cally generate diverse audio for image caption- ing datasets. This supports pretraining deep networks for encoding both audio and images, which we do via a dual encoder that learns to align latent representations from both modali- ties. We show that a masked margin softmax loss for such models is superior to the stan- dard triplet loss. We fine-tune these models on the Flickr8k Audio Captions Corpus and ob- tain state-of-the-art results—improving recall in the top 10 from 29.6% to 49.5%. We also obtain human ratings on retrieval outputs to better assess the impact of incidentally match- ing image-caption pairs that were not associ- ated in the data, finding that automatic evalua- tion substantially underestimates the quality of the retrieved results. 1 Introduction Natural language learning in people starts with speech, not text. Text is tidy: it comes in con- venient symbolic units that vary little from one writer to another. Speech is continuous and messy: the sounds used to convey a given word are modi- fied by those of surrounding words, and the rate of speech, its pitch, and more vary across speakers and even for the same speaker in different contexts. As such, problems involving speech provide distinct challenges and opportunities for learning language representations that text-based work—which repre- sents the vast majority—gets a free pass on. Recent work has explored various means to trans- form raw speech into symbolic forms with little or no supervision (Park and Glass, 2007; Varadara- jan et al., 2008; Ondel et al., 2016; Kamper et al., ∗ Work done as a member of the Google AI Residency Program. Figure 1: Models that encode speech segments and im- ages into a shared latent space enable images to be re- trieved using their audio descriptions (top) and to asso- ciate images with spoken captions (bottom). Text cap- tions are provided for clarity; only speech and images are used by the models. 2017a; Bhati et al., 2018). However, learning natu- ral language starts with grounded, contextualized speech. While infants as young as 8-months-old can segment word-like units without non-linguistic information (Jusczyk and Aslin, 1995) and adults can learn to segment words in artificial languages (Saffran et al., 1996), a learner must ultimately ground their representations of linguistic sequences (Harnad, 1990) to effectively use them to refer to objects, events and more. Furthermore, learning from rich perceptual data and interactions can be more efficient as it provides additional cues to the identities of words and their meaning in context. …

Content truncated to 500 words