MLMLM: Link Prediction with Mean Likelihood Masked Language Model

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4321–4331 August 1–6, 2021. ©2021 Association for Computational Linguistics 4321 MLMLM: Link Prediction with Mean Likelihood Masked Language Model Louis Clouatre∗†, Philippe Trempe∗, Amal Zouaq∗, Sarath Chandar∗ † ∇ ∗ Polytechnique Montr´eal † Mila - Quebec AI Institute ∇ Canada CIFAR AI Chair {philippe.trempe,amal.zouaq,sarath.chandar}@polymtl.ca {clouatrl,sarath.chandar}@mila.quebec.ca Abstract Knowledge Bases (KBs) are easy to query, ve- rifiable, and interpretable. They however scale with man-hours and high-quality data. Masked Language Models (MLMs), such as BERT, scale with computing power as well as unstruc- tured raw text data. The knowledge contained within these models is however not directly in- terpretable. We propose to perform link pre- diction with MLMs to address both the KBs scalability issues and the MLMs interpretabil- ity issues. By committing the knowledge em- bedded in MLMs to a KB, it becomes inter- pretable. To do that we introduce MLMLM, Mean Likelihood Masked Language Model, an approach comparing the mean likelihood of generating the different entities to perform link prediction in a tractable manner. We obtain State of the Art (SotA) results on the WN18RR dataset and SotA results on the Precision@1 metric on the WikidataM5 inductive and trans- ductive setting. We also obtain convincing re- sults on link prediction on previously unseen entities, making MLMLM a suitable approach to introducing new entities to a KB. 1 Introduction 1.1 Context KBs have many desirable properties. They are easy to query, verifiable, and perhaps most importantly interpretable by humans. They however have one critical shortcoming, they are expensive to build, making them harder to scale. Indeed, modern KBs scale with high-quality data, manual labor, or a mix of both. Approaches that scale with available computation and the massive amounts of unstruc- tured data that are being created and accumulated have proven invaluable in the recent deep learning boom. Large pretrained MLMs (Devlin et al., 2018; Liu et al., 2019) have been shown to scale well with large amounts of unstructured text data as well as with computing power. They also have dis- played some interesting emergent abilities, such as the ability to perform zero-shot question answering (Radford et al., 2019; Brown et al., 2020). This ability implies that the model parameters contain a large amount of factual knowledge that it can lever- age to answer a wide variety of questions. However, that knowledge is hardly interpretable by humans, as it is hidden within the millions to billions of parameters of the language model. By using MLMs to completes KBs, we can ad- dress both the issue of scalability of KBs and the is- sue of the interpretability of MLMs by committing knowledge of the latter to an interpretable format in the former. The MLM can learn new knowledge from the large amount of unstructured textual data that keeps being added to the World Wide Web and then be used to continually complete and update the KB. This has the very desirable effect of making the link prediction approach scale with both compu- …

Content truncated to 500 words