Fine-Grained Grounding for Multimodal Speech Recognition

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2667–2677 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2667 Fine-Grained Grounding for Multimodal Speech Recognition Tejas Srinivasan Language Technologies Institute Carnegie Mellon University tsriniva@andrew.cmu.edu Ramon Sanabria CSTR, ILCC University of Edinburgh r.sanabria@ed.ac.uk Florian Metze Language Technologies Institute Carnegie Mellon University fmetze@andrew.cmu.edu Desmond Elliott Department of Computer Science University of Copenhagen de@di.ku.dk Abstract Multimodal automatic speech recognition sys- tems integrate information from images to im- prove speech recognition quality, by ground- ing the speech in the visual context. While visual signals have been shown to be useful for recovering entities that have been masked in the audio, these models should be capable of recovering a broader range of word types. Existing systems rely on global visual features that represent the entire image, but localizing the relevant regions of the image will make it possible to recover a larger set of words, such as adjectives and verbs. In this paper, we pro- pose a model that uses finer-grained visual in- formation from different parts of the image, using automatic object proposals. In experi- ments on the Flickr8K Audio Captions Cor- pus, we find that our model improves over ap- proaches that use global visual features, that the proposals enable the model to recover enti- ties and other related words, such as adjectives, and that improvements are due to the model’s ability to localize the correct proposals.1 1 Introduction Multimodal language processing is inspired by evidence that conceptual representations in hu- mans are distributed across modality-specific sys- tems (Barsalou, 2003). In recent years, researchers have developed deep learning models that combine visual, linguistic, and auditory modalities for a va- riety of multimodal tasks, such as automatic image captioning (Vinyals et al., 2015), visual question- answering (Antol et al., 2015), and image–speech retrieval (Harwath and Glass, 2015), inter-alia. In multimodal automatic speech recognition (ASR), there have been efforts to integrate visual context into acoustic models (Miao and Metze, 2016) and sequence-to-sequence models (Palaskar 1The code is available at https://github.com/ tejas1995/MultimodalASR A black dog in a purple collar running Decoder purple collar in dog black a running Audio Encoder a Masked Speech Figure 1: Our multimodal speech recognition model transcribes masked speech using visual features ex- tracted from object proposals. et al., 2018; Sanabria et al., 2018; Caglayan et al., 2019). However, it is not clear if the visual context actually improves ASR or if it helps to regularize the model (Caglayan et al., 2019). Srinivasan et al. (2020) recently showed that global visual context (a single feature vector representing the entire image) is useful when the visually depictable linguistic inputs are masked, i.e., masking the speech that refer to entities. This experimental methodology, inspired by Caglayan et al. (2019), creates a system- atic gap in the speech signal that can be resolved by leveraging the visual context; for example, when the audio drops during online distance-based learn- ing or video calls with family and friends. We present a model for multimodal …

Content truncated to 500 words