Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs

Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 39–54 September 12, 2023 ©2023 Association for Computational Linguistics Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs Agnes Axelsson and Gabriel Skantze Division of Speech, Music and Hearing (TMH) / Lindstedtsvägen 24 KTH Royal Institute of Technology / Stockholm, Sweden agnaxe@kth.se and skantze@kth.se Abstract In any system that uses structured knowledge graph (KG) data as its underlying knowledge representation, KG-to-text generation is a use- ful tool for turning parts of the graph data into text that can be understood by humans. Re- cent work has shown that models that make use of pretraining on large amounts of text data can perform well on the KG-to-text task, even with relatively little training data on the specific graph-to-text task. In this paper, we build on this concept by using large language models to perform zero-shot generation based on noth- ing but the model’s understanding of the triple structure from what it can read. We show that ChatGPT achieves near state-of-the-art perfor- mance on some measures of the WebNLG 2020 challenge, but falls behind on others. Addition- ally, we compare factual, counter-factual and fictional statements, and show that there is a significant connection between what the LLM already knows about the data it is parsing and the quality of the output text. 1 Introduction For any system that presents verbal information to users, whether that information is in the form of text or audio, it can be useful to generate the system’s speech or text from a consistent underly- ing knowledge representation based on structured data. A commonly used data representation is knowledge graphs (KGs), where information is stored as properties or relations tying entities to- gether (Hogan et al., 2021). The combination of a property, its source and its target is referred to as a triple (Lassila and Swick, 1999; Hogan et al., 2021). KGs as an underlying structured data rep- resentation have been used to allow systems to tell narrative information (Colas et al., 2022), to re- trieve information in chatbots (Zhou et al., 2020) or recommender systems (Shao et al., 2021), and to reason about the grounding status of the infor- mation in terms of what the user currently knows (Axelsson and Skantze, 2020). Traditionally, template-based approaches for generating text from knowledge graphs have been sufficient for confined dialogue domains (Konstas and Lapata, 2013; Duma and Klein, 2013; Perera and Nand, 2015). An alternative is to train a data- driven end-to-end generation model, but a limiting factor is the relative lack of human-labelled data for the task. The WebNLG datasets produced for the challenges in 2017 (Gardent et al., 2017a) and 2020 (Castro Ferreira et al., 2020) are relatively small, and although recent progress has been made on producing much larger datasets (Colas et al., 2022), methods for natural language generation from knowledge graphs have generally had to work around the absence of large datasets. Recently, approaches that use …

Content truncated to 500 words