MultiQT: Multimodal learning for real-time question tracking in speech

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2370–2380 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2370 MultiQT: Multimodal Learning for Real-Time Question Tracking in Speech Jakob D. Havtorn Jan Latko Joakim Edin Lasse Borgholt Lars Maaløe Lorenzo Belgrano Nicolai F. Jacobsen Regitze Sdun ˇZeljko Agi´c Corti Store Strandstræde 21, 4 1255 Copenhagen K, Denmark jdh@corti.ai Abstract We address a challenging and practical task of labeling questions in speech in real time during telephone calls to emergency medi- cal services in English, which embeds within a broader decision support system for emer- gency call-takers. We propose a novel mul- timodal approach to real-time sequence label- ing in speech. Our model treats speech and its own textual representation as two separate modalities or views, as it jointly learns from streamed audio and its noisy transcription into text via automatic speech recognition. Our re- sults show significant gains of jointly learn- ing from the two modalities when compared to text or audio only, under adverse noise and lim- ited volume of training data. The results gen- eralize to medical symptoms detection where we observe a similar pattern of improvements with multimodal learning. 1 Introduction Our paper addresses the challenge of learning to discover and label questions in telephone calls to emergency medical services in English. The task is demanding in two key aspects: 1. Noise: A typical phone call to an emergency medical service differs significantly from data within most standard speech datasets. Most im- portantly, emergency calls are noisy by nature due to very stressful conversations conveyed over poor telephone lines. Automatic speech recogni- tion (ASR) and subsequent text processing quickly becomes prohibitive in such noisy environments, where word error rates (WER) are significantly higher than for standard benchmark data (Han et al., 2017). For this reason, we propose a sequence la- beler that makes use of two modalities of a phone call: audio and its transcription into text by utiliz- ing an ASR model. Hereby we create a multimodal Figure 1: A speech sequence from our phone call dataset. Two audio segments are highlighted: a ques- tion (in blue) and a reported symptom (in yellow). architecture that is more robust to the adverse con- ditions of an emergency call. 2. Real-time processing: Our model is required to work incrementally to discover questions in real time within incoming streams of audio in order to work as a live decision support system. At runtime, no segmentation into sub-call utterances such as phrases or sentences is easily available. The lack of segmentation coupled with the real-time pro- cessing constraint makes it computationally pro- hibitive to discover alignments between speech and its automatic transcription. For these reasons, we cannot utilize standard approaches to multimodal learning which typically rely on near-perfect cross- modal alignments between short and well-defined segments (Baltruˇsaitis et al., 2018). Context and relevance. Learning to label se- quences of text is one of the more thoroughly explored topics in natural language processing. In recent …

Content truncated to 500 words