SKILL: Structured Knowledge Infusion for Large Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1581 - 1588 July 10-15, 2022 ©2022 Association for Computational Linguistics SKILL: Structured Knowledge Infusion for Large Language Models Fedor Moiseev∗ 1,2 Zhe Dong† 2 Enrique Alfonseca 2 Martin Jaggi 1 1EPFL, Switzerland 2Google, Switzerland {femoiseev, zhedong, ealfonseca}@google.com, martin.jaggi@epfl.ch Abstract Large language models (LLMs) have demon- strated human-level performance on a vast spec- trum of natural language tasks. However, it is largely unexplored whether they can better internalize knowledge from a structured data, such as a knowledge graph, or from text. In this work, we propose a method to infuse structured knowledge into LLMs, by directly training T5 models on factual triples of knowledge graphs (KGs). We show that models pre-trained on Wikidata KG with our method outperform the T5 baselines on FreebaseQA and WikiHop, as well as the Wikidata-answerable subset of Triv- iaQA and NaturalQuestions. The models pre- trained on factual triples compare competitively with the ones on natural language sentences that contain the same knowledge. Trained on a smaller size KG, WikiMovies, we saw 3× improvement of exact match score on MetaQA task compared to T5 baseline. The proposed method has an advantage that no alignment be- tween the knowledge graph and text corpus is required in curating training data. This makes our method particularly useful when working with industry-scale knowledge graphs. 1 Introduction Large pre-trained language models, such as BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2019), REALM (Guu et al., 2020) and ERNIE (Sun et al., 2021) have become the state-of-the-art technology for many tasks. They are commonly pre-trained using unstructured text corpora, on tasks such as next word prediction, next sentence prediction (NSP) or masked lan- guage modelling (MLM). Especially for T5, self- supervised learning on unlabelled text corpus with MLM has been a common pre-training recipe (Roberts et al., 2020). This is normally followed by a fine-tuning step on the task of interest (Ruder ∗Work done during internship at Google. †Correspondence Author. et al., 2019), although large language models have also proved useful without this task-specific fine- tuning (Brown et al., 2020). Beyond the capacity of contextual understand- ing, human-level language understanding pivots on the knowledge about the world. The world knowl- edge is often expressed as factual triples (c.f. Ji et al., 2020), in the form of (subject entity, relation, object entity). A knowledge graph (KG) defined by a set of factual triples consists of the subjects and objects as vertices/nodes, and the relations form- ing the edges connecting them. Most of the large scale KGs (e.g. Wikidata, Vrandeˇci´c and Krötzsch, 2014) are stored in triple format. LLMs demonstrate some capacity of learning world knowledge from the natural text corpus (Roberts et al., 2020), but it is unclear to what degree they are also able to learn and memorize new knowledge directly from structured KG triples, or from text describing them explicitly. In order to infuse …

Content truncated to 500 words