Hybrid Attention based Multimodal Network for Spoken Language Classification

Proceedings of the 27th International Conference on Computational Linguistics, pages 2379–2390 Santa Fe, New Mexico, USA, August 20-26, 2018. 2379 Hybrid Attention based Multimodal Network for Spoken Language Classification Yue Gu, Kangning Yang*, Shiyu Fu*, Shuhong Chen, Xinyu Li and Ivan Marsic Multimedia Image Processing Lab Electrical and Computer Engineering Department Rutgers University, Piscataway, NJ, USA {yue.guapp, ky189, sf568, sc1624, Xinyu.li1118, marsic}@rutgers.edu Abstract We examine the utility of linguistic content and vocal characteristics for multimodal deep learn- ing in human spoken language understanding. We present a deep multimodal network with both feature attention and modality attention to classify utterance-level speech data. The proposed hy- brid attention architecture helps the system focus on learning informative representations for both modality-specific feature extraction and model fusion. The experimental results show that our system achieves state-of-the-art or competitive results on three published multimodal datasets. We also demonstrated the effectiveness and generalization of our system on a medical speech dataset from an actual trauma scenario. Furthermore, we provided a detailed comparison and analysis of traditional approaches and deep learning methods on both feature extraction and fu- sion. 1 Introduction Understanding human conversation is fundamental for human-computer interaction (HCI) and artificial intelligence (AI). However, it is hard for a computer to precisely interpret human meaning because: 1. Giving computers the ability to interpret speech requires a complete understanding of how it works for a human. Unfortunately, we still cannot identify how humans understand the information during conver- sation. 2. It is hard to extract associated features; there is a gap between the extracted modality-specific features and the actual human state of mind. 3. During conversation, people often accept messages from multiple sources such as facial expression, gesture, linguistic content, and vocal signals. But how to integrate the heterogeneous inputs into a computer is still an open-ended question. In this paper, we focus on exploring and addressing the issues above on the spoken language classification task, which is the most commonly used method for human sentiment analysis, emotion recognition, and speech topic categorization. Comprehending spoken language can be rephrased as analyzing vocal signals. A variety of research focuses on extracting the human meaning from audio data (Koolagudi et al., 2012; Busso et al., 2013; Mirsamadi et al., 2017). However, instead of directly processing audio information at the frame-level as a computer does, humans comprehend the meaning of utterances on the word-level, which can be seen as extracting meaning from linguistic content. It is natural for beginners in English to translate speech information into their native language word-by-word to understand the conversation, effectively using textual information for spoken language understanding. Based on this assumption, we propose a multimodal structure that considers both the audio signal and text as inputs to classify utterance-level speech data, where text data can be either transcribed speech or automatic speech-to-text output. Another challenge for classifying spoken language is extracting the associated features. As mentioned in previous research (Poria et al., 2017a), traditional approaches used handcrafted features for both the text and audio branches, …

Content truncated to 500 words