Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6452–6462 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6452 Symbolic inductive bias for visually grounded learning of spoken language Grzegorz Chrupała Tilburg University g.chrupala@uvt.nl Abstract A widespread approach to processing spo- ken language is to first automatically tran- scribe it into text. An alternative is to use an end-to-end approach: recent works have pro- posed to learn semantic embeddings of spoken language from images with spoken captions, without an intermediate transcription step. We propose to use multitask learning to exploit ex- isting transcribed speech within the end-to-end setting. We describe a three-task architecture which combines the objectives of matching spoken captions with corresponding images, speech with text, and text with images. We show that the addition of the SPEECH/TEXT task leads to substantial performance improve- ments on image retrieval when compared to training the SPEECH/IMAGE task in isolation. We conjecture that this is due to a strong in- ductive bias transcribed speech provides to the model, and offer supporting evidence for this. 1 Introduction Understanding spoken language is one of the key capabilities of intelligent systems which need to interact with humans. Applications include per- sonal assistants, search engines, vehicle naviga- tion systems and many others. The standard ap- proach to understanding spoken language both in industry and in research has been to decompose the problem into two components arranged in a pipeline: Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). The audio signal representing a spoken utterance is first transcribed into written text, which is subse- quently processed to extract some semantic rep- resentation of the utterance. Recent works have proposed to learn semantic embeddings of spo- ken language by using photographic images of ev- eryday situations matched with their spoken cap- tions, without an intermediate transcription step (Harwath et al., 2016; Chrupała et al., 2017). The weak and noisy supervision in these approaches is closer to how humans learn to understand speech by grounding it in perception and thus more useful as a cognitive model. It can also have some practi- cal advantages: in certain circumstances it may be easier to find or collect speech associated with im- ages rather than transcribed speech – for example when dealing with language whose speakers are il- literate, or for languages with no standard writing system (note that even some languages with many millions of speakers, like Cantonese, may not have a standardized writing system). On the other hand, the learning problem in this type of framework is less constrained, and harder, than standard ASR. In order to alleviate this shortcoming, we pro- pose to use multitask learning (MTL) and exploit transcribed speech within the end-to-end visually- grounded setting, and thus combine some features of both the pipeline and end-to-end approaches. Incorporating speech transcriptions into the end- to-end architecture via multi-task learning measn that the amount of transcribed speech and its qual- ity do not need to be as high as …

Content truncated to 500 words