Integrating Multimodal Information in Large Pretrained Transformers

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2359–2369 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2359 Integrating Multimodal Information in Large Pretrained Transformers Wasifur Rahman1, Md. Kamrul Hasan1*, Sangwu Lee1*, Amir Zadeh2, Chengfeng Mao2, Louis-Philippe Morency2, Ehsan Hoque1 1 - Department of Computer Science, University of Rochester, USA 2 - Language Technologies Institute, SCS, CMU, USA echowdh2@ur.rochester.edu, mhasan8@cs.rochester.edu, slee232@u.rochester.edu,abagherz@cs.cmu.edu, chengfem@andrew.cmu.edu,morency@cs.cmu.edu, mehoque@cs.rochester.edu Abstract Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving supe- rior performance downstream. While fine- tuning these pre-trained models is straight- forward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face com- munication). Pre-trained models don’t have the necessary components to accept two ex- tra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adapta- tion Gate (MAG). MAG allows BERT and XL- Net to accept multimodal nonverbal data dur- ing fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the vi- sual and acoustic modalities. In our experi- ments, we study the commonly used CMU- MOSI and CMU-MOSEI datasets for multi- modal sentiment analysis. Fine-tuning MAG- BERT and MAG-XLNet significantly boosts the sentiment analysis performance over pre- vious baselines as well as language-only fine- tuning of BERT and XLNet. On the CMU- MOSI dataset, MAG-XLNet achieves human- level multimodal sentiment analysis perfor- mance for the first time in the NLP commu- nity. 1 Introduction Human face-to-face communication flows as a seamless integration of language, acoustic, and vi- sion modalities. In ordinary everyday interactions, we utilize all these modalities jointly to convey our * - Equal contribution intentions and emotions. Understanding this face- to-face communication falls within an increasingly growing NLP research area called multimodal lan- guage analysis (Zadeh et al., 2018b). The biggest challenge in this area is to efficiently model the three pillars of communication together. This gives artificial intelligence systems the capability to com- prehend the multi-sensory information without dis- regarding nonverbal factors. In many applications such as dialogue systems and virtual reality, this capability is crucial to maintain the high quality of user interaction. The recent success of contextual word rep- resentations in NLP is largely credited to new Transformer-based (Vaswani et al., 2017) models such as BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019). These Transformer-based mod- els have shown performance improvement across downstream tasks (Devlin et al., 2018). However, their true downstream potential comes from fine- tuning their pre-trained models for particular tasks (Devlin et al., 2018). This is often done easily for lexical datasets which exhibit language modality only. However, this fine-tuning for multimodal language is neither trivial nor yet studied; simply because both BERT and XLNet only expect lin- …

Content truncated to 500 words