Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554–3565 June 6–11, 2021. ©2021 Association for Computational Linguistics 3554 Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training Oshin Agarwal∗1 Heming Ge2 Siamak Shakeri2 Rami Al-Rfou2 1 University of Pennsylvania 2Google Research oagarwal@seas.upenn.edu, {hemingge, siamaks, rmyeid}@google.com Abstract Prior work on Data-To-Text Generation, the task of converting knowledge graph (KG) triples into natural text, focused on domain- specific benchmark datasets. In this paper, however, we verbalize the entire English Wiki- data KG, and discuss the unique challenges associated with a broad, open-domain, large- scale verbalization. We further show that ver- balizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate struc- tured KGs and natural language corpora. In contrast to the many architectures that have been developed to integrate these two sources, our approach converts the KG into natural text, allowing it to be seamlessly integrated into existing language models. It carries the fur- ther advantages of improved factual accuracy and reduced toxicity in the resulting language model. We evaluate this approach by augment- ing the retrieval corpus in a retrieval language model and showing significant improvements on the knowledge intensive tasks of open do- main QA and the LAMA knowledge probe. 1 Introduction Data-To-Text Generation (Kukich, 1983; Goldberg et al., 1994) involves converting knowledge graph (KG) triples of the form (subject, relation, object) into a natural language sentence(s). There are many standard datasets for this task such as WebNLG (Gardent et al., 2017) and many systems have been developed to improve performance on these datasets. However, to the best of our knowl- edge, no prior work has attempted to verbalize a full knowledge graph. Verbalizing a full KG has ad- ditional challenges over small benchmark datasets, such as entity and relation coverage and the lack of grouped sets of triples that can produce coher- ent sentences together. In this paper, we convert the English Wikidata KG (Vrandeˇci´c and Krötzsch, 2014) into natural language text (Figure 1). The ∗Work done during internship at Google Spork EP 1995 The Shins English language Extended play Spork EP is the English language extended play by the band The Shins. It was released in 1995 performer publication date Instance of language of work or name TEKGEN Figure 1: An example of generating text from KG. First, the entity subgraphs on the left are created and then converted to the sentence on the right. generated corpus, which we call the KELM Cor- pus, consists of ∼18M sentences spanning ∼45M triples with ∼1500 distinct relations. For training the verbalization system, we also create an English Wikidata KG–Wikipedia Text aligned corpus con- sisting of a variety of entities such as dates and numerical quantities. We evaluate the quality of the generated corpus through human evaluation of a random sample. We further showcase the utility of this corpus in lan- guage model pre-training. Text represents a limited coverage of the world knowledge. Therefore, we …

Content truncated to 500 words