Multimodal Transformer for Unaligned Multimodal Language Sequences

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558–6569 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6558 Multimodal Transformer for Unaligned Multimodal Language Sequences Yao-Hung Hubert Tsai†∗, Shaojie Bai†∗, Paul Pu Liang†, J. Zico Kolter†‡, Louis-Philippe Morency†, Ruslan Salakhutdinov† † Carnegie Mellon University, ‡ Bosch Center for AI {yaohungt,shaojieb,pliang,zkolter,morency,rsalakhu}@cs.cmu.edu Abstract Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. How- ever, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range depen- dencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner with- out explicitly aligning the data. At the heart of our model is the directional pairwise cross- modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experi- ments on both aligned and non-aligned multi- modal time-series show that our model outper- forms state-of-the-art methods by a large mar- gin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal atten- tion mechanism in MulT. 1 Introduction Human language possesses not only spoken words but also nonverbal behaviors from vision (facial attributes) and acoustic (tone of voice) modali- ties (Gibson et al., 1994). This rich information provides us the benefit of understanding human behaviors and intents (Manning et al., 2014). Nev- ertheless, the heterogeneities across modalities of- ten increase the difficulty of analyzing human lan- guage. For example, the receptors for audio and vision streams may vary with variable receiving frequency, and hence we may not obtain optimal mapping between them. A frowning face may re- late to a pessimistically word spoken in the past. ∗*equal contribution. It’s huge sort of spectacle movie Vision Language Audio [ ] [ [ [ [ [ ] ] ] ] ] [ [ [ [ [ [ ] ] ] ] ] ] … … … … … … … … … Vision-to-Language Alignment Audio-to-Language Alignment (Pre-defined Word-level) Alignment It’s huge sort of spectacle movie Vision Language Audio … … … … … … … … … Vision-to-Language (‘spectacle’) Attention Weights Audio-to-Language (‘spectacle’) Attention Weights (Ours) Crossmodal Attention (uninformative) (eyebrows raise) (emphasis) (emphasis) (neutral) Figure 1: Example video clip from movie reviews. [Top]: Illustration of word-level alignment where video and audio features are averaged across the time interval of each spoken word. [Bottom] Illustration of cross- modal attention weights between text (“spectacle”) and vision/audio. That is to say, multimodal language sequences often exhibit “unaligned” nature and require in- ferring long term dependencies across modalities, which raises a question on performing efficient multimodal fusion. To address the above issues, in this paper we propose the Multimodal Transformer (MulT), an end-to-end model that extends the standard Trans- former network …

Content truncated to 500 words