MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10527 - 10536 December 7-11, 2022 ©2022 Association for Computational Linguistics MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion Yu Zhao1 Xiangrui Cai1∗ Yike Wu2 Haiwei Zhang1 Ying Zhang3 Guoqing Zhao4 Ning Jiang4 1 College of Cyber Science, TKLNDST, Nankai University, Tianjin, China 2 School of Journalism and Communication, Nankai University, Tianjin, China 3 College of Computer Science, Nankai University, Tianjin, China 4 Mashang Consumer Finance Co, Ltd zhaoyu@dbis.nankai.edu.cn, {caixr,wuyike,zhhaiwei,yingzhang}@nankai.edu.cn Abstract Multimodal knowledge graph completion (MKGC) aims to predict missing entities in MKGs. Previous works usually share rela- tion representation across modalities. This re- sults in mutual interference between modali- ties during training, since for a pair of enti- ties, the relation from one modality probably contradicts that from another modality. Fur- thermore, making a unified prediction based on the shared relation representation treats the input in different modalities equally, while their importance to the MKGC task should be different. In this paper, we propose MoSE, a Modality Split representation learning and Ensemble inference framework for MKGC. Specifically, in the training phase, we learn modality-split relation embeddings for each modality instead of a single modality-shared one, which alleviates the modality interference. Based on these embeddings, in the inference phase, we first make modality-split predictions and then exploit various ensemble methods to combine the predictions with different weights, which models the modality importance dy- namically. Experimental results on three KG datasets show that MoSE outperforms state-of- the-art MKGC methods. Codes are available at https://github.com/OreOZhao/MoSE4MKGC. 1 Introduction Multimodal knowledge graphs (MKGs) organize multimodal facts in the form of entities and rela- tions, and have been successfully applied to vari- ous knowledge-driven tasks (Marino et al., 2019; Sun et al., 2020; Zhang et al., 2018; Zhu et al., 2022). To address the inherent incomplete prob- lems in MKGs, multimodal knowledge graph com- pletion (MKGC) has been proposed (Xie et al., 2016, 2017), which utilizes auxiliary visual or text information to help predict missing entities. As shown in Figure 1a, given the head entity �� �� Friends ∗Corresponding author. I-TCR E-TCR TCR Decouple concat Friends The United States of America Country_of_Origin “Friends is an American sitcom created by David Crane and Marta Kauffman, …” TV/Tv_Program “ The United States of America, commonly referred to as the United States, America…” ✗ ✓ (a) An example of contradictory relations between modalities. I-TCR E-TCR TCR Decouple concat Friends The United States of America Country_of_Origin “Friends is an American sitcom created by David Crane and Marta Kauffman, …” TV/Tv_Program “ The United States of America, commonly referred to as the United States, America…” ✗ ✓ (b) Implicit (I-) and Explicit (E-) Tight-Coupling Relation (TCR), and TCR Decouple (Ours). Note that the arrows with the same color represent the same relation embedding. Figure 1: (a) displays an example of multimodal triples, in which text modality demonstrates relevant relation ✓ and visual modality demonstrates irrelevant relation ✗. (b) visualizes existing approaches with implicit and explicit …

Content truncated to 500 words