Multimodal and Multiresolution Speech Recognition with Transformers

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2381–2387 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2381 Multiresolution and Multimodal Speech Recognition with Transformers Georgios Paraskevopoulos Srinivas Parthasarathy Aparna Khare Shiva Sundaram Amazon Lab126 geopar@central.ntua.gr, {parsrini,apkhare,sssundar}@amazon.com Abstract This paper presents an audio visual automatic speech recognition (AV-ASR) system using a Transformer-based architecture. We particu- larly focus on the scene context provided by the visual information, to ground the ASR. We extract representations for audio features in the encoder layers of the transformer and fuse video features using an additional crossmodal multihead attention layer. Additionally, we in- corporate a multitask training criterion for mul- tiresolution ASR, where we train the model to generate both character and subword level tran- scriptions. Experimental results on the How2 dataset, indicate that multiresolution training can speed up convergence by around 50% and relatively improves word error rate (WER) per- formance by upto 18% over subword predic- tion models. Further, incorporating visual in- formation improves performance with relative gains upto 3.76% over audio only models. Our results are comparable to state-of-the-art Lis- ten, Attend and Spell-based architectures. 1 Introduction Automatic speech recognition is a fundamental technology used on a daily basis by millions of end-users and businesses. Applications include au- tomated phone systems, video captioning and voice assistants providing an intuitive and seemless in- terface between users and end systems. Current ASR approaches rely solely on utilizing audio in- put to produce transcriptions. However, the wide availability of cameras in smartphones and home devices acts as motivation to build AV-ASR models that rely on and benefit from multimodal input. Traditional AV-ASR systems focus on tracking the user’s facial movements and performing lipread- ing to augment the auditory inputs (Potamianos et al., 1997; Mroueh et al., 2015; Tao and Busso, 2018). The applicability of such models in real world environments is limited, due to the need for accurate audio-video alignment and careful camera placement. Instead, we focus on using video to contextualize the auditory input and perform multi- modal grounding. For example, a basketball court is more likely to include the term “lay-up” whereas an office place is more likely include the term “lay- off”. This approach can boost ASR performance, while the requirements for video input are kept relaxed (Caglayan et al., 2019; Hsu et al., 2019). Additionally we consider a multiresolution loss that takes into account transcriptions at the charac- ter and subword level. We show that this scheme regularizes our model showing significant improve- ments over subword models. Multitask learning on multiple levels has been previously explored in the literature, mainly in the context of CTC (Sanabria and Metze, 2018; Krishna et al., 2018; Ueno et al., 2018). A mix of seq2seq and CTC approaches combine word and character level (Kremer et al., 2018; Ueno et al., 2018) or utilize explicit phonetic information (Toshniwal et al., 2017; Sanabria and Metze, 2018). Modern ASR systems rely on end-to-end, align- ment free neural architectures, i.e. CTC (Graves et …

Content truncated to 500 words