Textual Supervision for Visually Grounded Spoken Language Understanding

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2698–2709 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2698 Textual Supervision for Visually Grounded Spoken Language Understanding Bertrand Higy Cognitive Science and AI Tilburg University b.j.r.higy@uvt.nl Desmond Elliott Department of Computer Science University of Copenhagen de@di.ku.dk Grzegorz Chrupała Cognitive Science and AI Tilburg University g.chrupala@uvt.nl Abstract Visually-grounded models of spoken language understanding extract semantic information di- rectly from speech, without relying on tran- scriptions. This is useful for low-resource lan- guages, where transcriptions can be expensive or impossible to obtain. Recent work showed that these models can be improved if transcrip- tions are available at training time. However, it is not clear how an end-to-end approach com- pares to a traditional pipeline-based approach when one has access to transcriptions. Com- paring different strategies, we find that the pipeline approach works better when enough text is available. With low-resource languages in mind, we also show that translations can be effectively used in place of transcriptions but more data is needed to obtain similar results. 1 Introduction Spoken language understanding promises to trans- form our interactions with technology by allowing people to control electronic devices through voice commands. However, mapping speech to meaning is far from trivial. The traditional approach, which has proven its effectiveness, relies on text as an in- termediate representation. This so-called pipeline approach combines an automatic speech recogni- tion (ASR) system and a natural language under- standing (NLU) component. While this allows us to take advantage of improvements achieved in both fields, it requires transcribed speech, which is an expensive resource. Visually-grounded models of spoken language understanding (Harwath et al., 2016; Chrupała et al., 2017) were recently introduced to extract semantic information from speech directly, without relying on textual information (see Figure 1 for an illustration). The advantages of these approaches are twofold: (i) expensive transcriptions are not necessary to train the system, which is beneficial A boy in a green shirt on a skateboard on a stone wall with graffiti. 落書きされた石の壁でスケートボードに乗っている緑色の シャツを着た少年。 Figure 1: An image described by an English spoken caption (represented by its spectrogram), its transcrip- tion, and translation into Japanese. Visually-grounded models are usually trained to map the image and its spoken caption into a shared semantic space. for low-resource languages and (ii) trained in an end-to-end fashion, the whole system is optimized for the final task, which has been shown to improve performance in other applications. While text is not necessary to train such systems, recent work has shown that they can greatly benefit from tex- tual supervision if available (Chrupała, 2019; Pasad et al., 2019), generally using the multitask learning setup (MTL) (Caruana, 1997). However, the end-to-end MTL-based models in previous works have not been compared against the more traditional pipeline approach that uses ASR as an intermediate step. The pipeline approach could be a strong baseline as, intuitively, written transcriptions are an accurate and concise represen- tation of spoken language. In this paper, we set out 2699 to …

Content truncated to 500 words