A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 225–234 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning Hatem Mousselly-Sergieh†, Teresa Botschen§†, Iryna Gurevych§†, Stefan Roth§‡ † Ubiquitous Knowledge Processing (UKP) Lab ‡ Visual Inference Lab § Research Training Group AIPHES Department of Computer Science, Technische Universit¨at Darmstadt h.m.sergieh@gmail.com {botschen@aiphes, gurevych@ukp.informatik, stefan.roth@visinf}.tu-darmstadt.de Abstract Current methods for knowledge graph (KG) representation learning focus solely on the structure of the KG and do not exploit any kind of external information, such as visual and linguistic information corresponding to the KG entities. In this paper, we propose a multimodal translation-based approach that defines the energy of a KG triple as the sum of sub-energy functions that leverage both mul- timodal (visual and linguistic) and structural KG representations. Next, a ranking-based loss is minimized using a simple neural net- work architecture. Moreover, we introduce a new large-scale dataset for multimodal KG representation learning. We compared the per- formance of our approach to other baselines on two standard tasks, namely knowledge graph completion and triple classification, using our as well as the WN9-IMG dataset.1 The results demonstrate that our approach outperforms all baselines on both tasks and datasets. 1 Introduction Knowledge Graphs (KGs), e.g., Freebase (Bol- lacker et al., 2008) and DBPedia (Auer et al., 2007), are stores of relational facts, which are crucial for various kinds of tasks, such as ques- tion answering and information retrieval. KGs are structured as triples of head and tail entities along with the relation that holds between them. Factual knowledge is virtually infinite and is frequently subject to change. This raises the question of the incompleteness of the KGs. To address this prob- lem, several methods have been proposed for au- tomatic KG completion (KGC, for a survey refer to Wang et al., 2017). In recent years, translation- based approaches have witnessed a great success. Their main idea is to model the entities and their 1Code and datasets are released for research purposes: https://github.com/UKPLab/ starsem18-multimodalKB relation as low-dimensional vector representations (embeddings), which in turn can be used to per- form different kinds of inferences on the KG. These include identifying new facts or validating existing ones. However, translation-based meth- ods rely on the rich structure of the KG and gener- ally ignore any type of external information about the included entities. In this paper, we propose a translation-based ap- proach for KG representation learning that lever- ages two different types of external, multimodal representations: linguistic representations created by analyzing the usage patterns of KG entities in text corpora, and visual representations obtained from images corresponding to the KG entities. To gain initial insights into the potential benefits of external information for the KGC task, let us con- sider the embeddings produced by the translation- based TransE method (Bordes et al., 2013) on the WN9-IMG dataset (Xie et al., 2017). This dataset contains a subset of WordNet synsets, which are linked according to a …

Content truncated to 500 words