Learning Word-Like Units from Joint Audio-Visual Analysis

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 506–517 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1047 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 506–517 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1047 Learning word-like units from joint audio-visual analysis David Harwath and James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139, USA {dharwath,glass}@mit.edu Abstract Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image re- gions. For example, our model is able to detect spoken instances of the words “lighthouse” within an utterance and as- sociate them with image regions contain- ing lighthouses. We do not use any form of conventional automatic speech recog- nition, nor do we use any text transcrip- tions or conventional linguistic annota- tions. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to rec- ognize word categories by sound, but also to enrich the words it learns with seman- tics by grounding them in images. 1 Introduction 1.1 Problem Statement and Motivation Automatically discovering words and other el- ements of linguistic structure from continuous speech has been a longstanding goal in com- putational linguists, cognitive science, and other speech processing fields. Practically all humans acquire language at a very early age, but this task has proven to be an incredibly difficult problem for computers. While conventional automatic speech recognition (ASR) systems have a long history and have recently made great strides thanks to the re- vival of deep neural networks (DNNs), their re- liance on highly supervised training paradigms has essentially restricted their application to the ma- jor languages of the world, accounting for a small fraction of the more than 7,000 human languages spoken worldwide (Lewis et al., 2016). The main reason for this limitation is the fact that these su- pervised approaches require enormous amounts of very expensive human transcripts. Moreover, the use of the written word is a convenient but limiting convention, since there are many oral languages which do not even employ a writing system. In constrast, infants learn to communicate verbally before they are capable of reading and writing - so there is no inherent reason why spoken language systems need to be inseparably tied to text. The key contribution of this paper has two facets. First, we introduce a methodology capable of not only discovering word-like units from con- tinuous speech at the waveform level with no ad- ditional text transcriptions or conventional speech recognition apparatus. Instead, we jointly learn the semantics of those units via visual associa- tions. Although we evaluate our algorithm on an English corpus, it could conceivably run on any language without requiring any text or associated ASR capability. Second, from a computational perspective, our method of …

Content truncated to 500 words