Text-Free Image-to-Speech Synthesis Using Learned Segmental Units

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5284–5300 August 1–6, 2021. ©2021 Association for Computational Linguistics 5284 Text-Free Image-to-Speech Synthesis Using Learned Segmental Units Wei-Ning Hsu1∗, David Harwath2∗, Tyler Miller2∗, Christopher Song3, James Glass1 1Massachusetts Institute of Technology 2University of Texas at Austin 3Johns Hopkins University wnhsu@csail.mit.edu, harwath@utexas.edu Abstract In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an inter- mediate representation or source of supervi- sion. Instead, we connect the image caption- ing module and the speech synthesis module with a set of discrete, sub-word speech units that are discovered with a self-supervised vi- sual grounding task. We conduct experiments on the Flickr8k spoken caption dataset in addi- tion to a novel corpus of spoken audio captions collected for the popular MSCOCO dataset, demonstrating that our generated captions also capture diverse visual semantics of the images they describe. We investigate several different intermediate speech representations, and em- pirically find that the representation must sat- isfy several important properties to serve as drop-in replacements for text. 1 Introduction Although there are over 7,000 languages spoken worldwide (Lewis et al., 2016), only several dozen have enough data available to support supervised speech recognition, and many languages do not even employ a writing system (Adda et al., 2016). In contrast, most people learn to use spoken lan- guage long before they learn to read and write, suggesting that linguistic annotation is not a pre- requisite for speech processing systems. This line of reasoning motivates research that aims to dis- cover meaningful linguistic abstractions (phones, words, etc.) directly from the speech signal, with the intention that they could reduce the reliance of spoken language systems on text transcripts. A rich body of work has recently emerged inves- tigating representation learning for speech using visual grounding objectives (Synnaeve et al., 2014; Harwath and Glass, 2015; Harwath et al., 2016; Kamper et al., 2017; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Alishahi et al., 2017; Scharenborg et al., 2018; Hsu and Glass, 2018a; Kamper et al., 2018; Surís et al., 2019; Il- harco et al., 2019; Eloff et al., 2019), as well as how word-like and subword-like linguistic units can be made to emerge within these models (Har- wath and Glass, 2017; Harwath et al., 2019; Drexler and Glass, 2017; Alishahi et al., 2017; Harwath et al., 2019; Harwath and Glass, 2019; Havard et al., 2019b; Harwath et al., 2020). So far, these efforts have predominantly focused on inference, where the goal is to learn a mapping from speech wave- forms to a semantic embedding space. Generation of speech conditioned on a point in a semantic space has been less explored, and is what we focus on in this work. We hypothesize that generative approaches offer interesting advantages over rely- ing solely on inference. For example, prior …

Content truncated to 500 words