Synchronous Dual Network with Cross-Type Attention for Joint Entity and Relation Extraction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 231–245 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 231 Improving Multimodal fusion via Mutual Dependency Maximisation Pierre Colombo1,2, Emile Chapuis1, Matthieu Labeau1, Chloe Clavel1 1LTCI, Telecom Paris, Institut Polytechnique de Paris, 2IBM GBS France, 1firstname.lastname@telecom-paris.fr, 2pierre.colombo@ibm.com, Abstract Multimodal sentiment analysis is a trending area of research, and the multimodal fusion is one of its most active topic. Acknowledg- ing humans communicate through a variety of channels (i.e visual, acoustic, linguistic), mul- timodal systems aim at integrating different unimodal representations into a synthetic one. So far, a consequent effort has been made on developing complex architectures allowing the fusion of these modalities. However, such sys- tems are mainly trained by minimising sim- ple losses such as L1 or cross-entropy. In this work, we investigate unexplored penalties and propose a set of new objectives that mea- sure the dependency between modalities. We demonstrate that our new penalties lead to a consistent improvement (up to 4.3 on accu- racy) across a large variety of state-of-the-art models on two well-known sentiment analysis datasets: CMU-MOSI and CMU-MOSEI. Our method not only achieves a new SOTA on both datasets but also produces representations that are more robust to modality drops. Finally, a by-product of our methods includes a statisti- cal network which can be used to interpret the high dimensional representations learnt by the model. 1 Introduction Humans employ three different modalities to com- municate in a coordinated manner: the language modality with the use of words and sentences, the vision modality with gestures, poses and facial ex- pressions and the acoustic modality through change in vocal tones. Multimodal representation learning has shown great progress in a large variety of tasks including emotion recognition, sentiment analy- sis (Soleymani et al., 2017), speaker trait analysis (Park et al., 2014) and fine-grained opinion min- ing (Garcia et al., 2019a). Learning from different modalities is an efficient way to improve perfor- mance on the target tasks (Xu et al., 2013). Never- theless, heterogeneities across modalities increase the difficulty of learning multimodal representa- tions and raise specific challenges. Baltrušaitis et al. (2018) identifies fusion as one of the five core challenges in multimodal representation learning, the four other being: representation, modality align- ment, translation and co-learning. Fusion aims at integrating the different unimodal representations into one common synthetic representation. Effec- tive fusion is still an open problem: the best multi- modal models in sentiment analysis (Rahman et al., 2020) improve over their unimodal counterparts, relying on text modality only, by less than 1.5% on accuracy. Additionally, the fusion should not only improve accuracy but also make representations more robust to missing modalities. Multimodal fusion can be divided into early and late fusion techniques: early fusion takes place at the feature level (Ye et al., 2017), while late fusion takes place at the decision or scoring level (Khan et al., 2012). Current research in multimodal sentiment analysis mainly focuses on developing new fusion mechanisms relying on deep …

Content truncated to 500 words