Multimodal Learning
Related Papers
Strong and Simple Baselines for Multimodal Utterance Embeddings
AbstractHuman language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due …...
Learning to Represent Image and Text with Denotation Graph
AbstractLearning to fuse vision and language information and representing them is an important research problem with many applications. Recent progresses have leveraged the ideas of pre-training (from language modeling) and …...
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
AbstractThe human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language. Recent multimodal learning with strong performances on …...
VolTAGE: Volatility Forecasting via Text Audio Fusion with Graph Convolution Networks for Earnings Calls
AbstractNatural language processing has recently made stock movement forecasting and volatility forecasting advances, leading to improved financial forecasting. Transcripts of companies’ earnings calls are well studied for risk modeling, offering …...
LT3 at SemEval-2021 Task 6: Using Multi-Modal Compact Bilinear Pooling to Combine Visual and Textual Understanding in Memes
AbstractInternet memes have become ubiquitous in social media networks today. Due to their popularity, they are also a widely used mode of expression to spread disinformation online. As memes consist …...
Multimodal Graph-based Transformer Framework for Biomedical Relation Extraction
The recent advancement of pre-trained Trans- former models has propelled the development of effective text mining models across various biomedical tasks. However, these models are primarily learned on the textual …...
On the (In)Effectiveness of Images for Text Classification
AbstractImages are core components of multi-modal learning in natural language processing (NLP), and results have varied substantially as to whether images improve NLP tasks or not. One confounding effect has …...
Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions
AbstractAs multimodal learning finds applications in a wide variety of high-stakes societal tasks, investigating their robustness becomes important. Existing work has focused on understanding the robustness of vision-and-language models to …...
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction
AbstractThe recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language …...