Dynamic Regularization in UDA for Transformers in Multimodal Classification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8700–8711 July 9-14, 2023 ©2023 Association for Computational Linguistics Dynamic Regularization in UDA for Transformers in Multimodal Classification Ivonne Monter-Aldana*, A. Pastor López-Monroy*, and Fernando Sánchez-Vega*, † * Department of Computer Science, Mathematics Research Center (CIMAT), Gto. Mexico. † Consejo Nacional de Ciencia y Tecnología (CONACyT), CDMX, México. {ivonne.monter, pastor.lopez, fernando.sanchez}@cimat.mx Abstract Multimodal machine learning is a cutting-edge field that explores ways to incorporate infor- mation from multiple sources into models. As more multimodal data becomes available, this field has become increasingly relevant. This work focuses on two key challenges in multi- modal machine learning. The first is finding efficient ways to combine information from different data types. The second is that often, one modality (e.g., text) is stronger and more relevant, making it difficult to identify mean- ingful patterns in the weaker modality (e.g., image). Our approach focuses on more ef- fectively exploiting the weaker modality while dynamically regularizing the loss function. First, we introduce a new two-stream model called Multimodal BERT-ViT, which features a novel intra-CLS token fusion. Second, we utilize a dynamic adjustment that maintains a balance between specialization and general- ization during the training to avoid overfitting, which we devised. We add this dynamic ad- justment to the Unsupervised Data Augmen- tation (UDA) framework. We evaluate the ef- fectiveness of these proposals on the task of multi-label movie genre classification using the Moviescope and MM-IMDb datasets. The evaluation revealed that our proposal offers substantial benefits, while simultaneously en- abling us to harness the weaker modality with- out compromising the information provided by the stronger. 1 Introduction Multimodal machine learning focuses on meth- ods for modeling information from more than one modality, such as text, image, audio, and video. It is crucial for models to simultane- ously analyze and organize diverse modalities. Recent advancements in multimodal applications have primarily been attributed to the availability of new large-scale multimodal datasets, increased computational capacity, and enhanced represen- tations of individual modalities. This develop- ment encompasses a broad scope, for example, multimodal classification (Arevalo et al., 2017; Cascante-Bonilla et al., 2019), gesture recognition (Yu et al., 2021), and audio-video clustering (Al- wassel et al., 2019). Multimodal machine learning is a great chal- lenge due to the multiple factors involved. In this work, we focus on multi-label movie genre classi- fication utilizing two modalities: text and image. We focus on addressing four main difficulties: (i) Models trained on different modalities learn and generalize at different speeds (Wang et al., 2020), and the optimization process benefits one modality more than the other. The consequence is that during the training phase, the model gives more weight to the most relevant (stronger) modal- ity and disregards the less informative (weaker) one. (ii) There is a lack of effective fusion since prediction by fusion modalities depends on the correlation between modalities and their represen- tation. (iii) Complex deep neural architectures re- quire a large computing capacity, and most …

Content truncated to 500 words