Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1946 - 1957 December 7-11, 2022 ©2022 Association for Computational Linguistics Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent? Pradip Pramanick and Chayan Sarkar Robotics & Autonomous Systems TCS Research, India {pradip.pramanick,sarkar.chayan}@tcs.com The usage of automatic speech recognition (ASR) systems are becoming omnipresent rang- ing from personal assistant to chatbots, home, and industrial automation systems, etc. Modern robots are also equipped with ASR capabilities for inter- acting with humans as speech is the most natural interaction modality. However, ASR in robots faces additional challenges as compared to a personal as- sistant. Being an embodied agent, a robot must recognize the physical entities around it and there- fore reliably recognize the speech containing the description of such entities. However, current ASR systems are often unable to do so due to limita- tions in ASR training, such as generic datasets and open-vocabulary modeling. Also, adverse condi- tions during inference, such as noise, accented, and far-field speech makes the transcription inaccurate. In this work, we present a method to incorporate a robot’s visual information into an ASR system and improve the recognition of a spoken utterance containing a visible entity. Specifically, we propose a new decoder biasing technique to incorporate the visual context while ensuring the ASR output does not degrade for incorrect context. We achieve a 59% relative reduction in WER from an unmodi- fied ASR system. 1 Introduction Spoken interaction with a robot not only increases its usability and acceptability, it provides a natural mode of interaction even for a novice user. The recent development of deep learning-based end-to- end automatic speech recognition (ASR) systems has achieved a very high accuracy (Li, 2021) as compared to traditional ASR systems. As a result, we see a huge surge of speech-based interfaces for many systems including robots. However, the accu- racy of any state-of-the-art ASR gets significantly impacted based on the dialect of the speaker, dis- tance of the speaker from the microphone, ambient noise, etc., particularly for novel and low-frequency take the bink illow Acoustic model Task understanding & grounded planning Object detector a black lamp on a table the sofa is black a pink pillow on the sofa Task planning failed (take the pink pillow) Figure 1: A simple pipeline of speech interface for human-robot interaction. take the pink pillow a black lamp on a table the sofa is black a pink pillow on the sofa 1. MOVE_TO sofa 2. LOCALIZE pink-pillow 3. PICK_UP pink-pillow Acoustic model Task understanding & grounded planning (take the pink pillow) Object detector Shallow fusion biasing Dynamic context extraction biasing Vocabulary [ black, lamp, table, sofa, black, pink, pillow ] Figure 2: Robust Speech Interface (RoSI) for embod- ied agents with shallow fusion biasing using dynamic biasing vocabulary. vocabularies. These factors are often predominant in many robotic applications. This not only results in poor translation accuracy but also impacts the instruction understanding and task execution capa- bility of the robot. …

Content truncated to 500 words