Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14734–14751 July 9-14, 2023 ©2023 Association for Computational Linguistics Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling Shengqiong Wu1, Hao Fei1∗, Yixin Cao2, Lidong Bing3, Tat-Seng Chua1 1 Sea-NExT Joint Lab, School of Computing, National University of Singapore 2 Singapore Management University, 3 DAMO Academy, Alibaba Group swu@u.nus.edu haofei37@nus.edu.sg caoyixin2011@gmail.com l.bing@alibaba-inc.com dcscts@nus.edu.sg Abstract Existing research on multimodal relation ex- traction (MRE) faces two co-existing chal- lenges, internal-information over-utilization and external-information under-exploitation. To combat that, we propose a novel frame- work that simultaneously implements the idea of internal-information screening and external- information exploiting. First, we represent the fine-grained semantic structures of the input im- age and text with the visual and textual scene graphs, which are further fused into a unified cross-modal graph (CMG). Based on CMG, we perform structure refinement with the guid- ance of the graph information bottleneck prin- ciple, actively denoising the less-informative features. Next, we perform topic modeling over the input image and text, incorporating latent multimodal topic features to enrich the contexts. On the benchmark MRE dataset, our system outperforms the current best model significantly. With further in-depth analyses, we reveal the great potential of our method for the MRE task. Our codes are open at https://github.com/ChocoWu/MRE-ISE. 1 Introduction Relation extraction (RE), determining the semantic relation between a pair of subject and object entities in a given text (Yu et al., 2020), has played a vital role in many downstream natural language process- ing (NLP) applications, e.g., knowledge graph con- struction (Wang et al., 2019; Mondal et al., 2021), question answering (Cao et al., 2022). But in real- istic scenarios (i.e., social media), data is often in various forms and modalities (i.e., texts, images), rather than pure texts. Thus, multimodal relation extraction has been introduced recently (Zheng et al., 2021b), where additional visual sources are added to the textual RE as an enhancement to the relation inference. The essence of a successful MRE lies in the effective utilization of multimodal ∗Corresponding author: Hao Fei Input Text: Input Image: Congratulations to Angela and Mark Salmons, a new life ahead is waiting! couple Yessir dropping my first single "Hot summer" with my brothers Migos. Useful feature for relation reasoning Input Text: Input Image: present in tour, video, billboard, concert, album, live … Topic: #Music ➢Example #1 ➢Example #2 Figure 1: Examples of multimodal relation extraction (MRE). In texts, the subject and object entities are un- derlined, and linked by the relational tags. information. Certain efforts have been made in existing MRE work and achieved promising per- formances, where delicate interaction and fusion mechanisms are designed for encoding the multi- modal features (Zheng et al., 2021a; Chen et al., 2022b,a). Nevertheless, current methods still fail to sufficiently harness the feature sources from two information perspectives, which may hinder further task development. Internal-information over-utilization. On the one hand, most existing MRE methods progres- sively incorporate full-scale textual …

Content truncated to 500 words