Evaluating Generative Models for Graph-to-Text Generation

Proceedings of Recent Advances in Natural Language Processing, pages 1256–1264 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_133 1256 Evaluating Generative Models for Graph-to-Text Generation Shuzhou Yuan, Michael Färber Karlsruhe Institute of Technology (KIT) {shuzhou.yuan, michael.faerber}@kit.edu Abstract Large language models (LLMs) have been widely employed for graph-to-text generation tasks. However, the process of finetuning LLMs requires significant training resources and annotation work. In this paper, we explore the capability of generative models to gener- ate descriptive text from graph data in a zero- shot setting. Specifically, we evaluate GPT- 3 and ChatGPT on two graph-to-text datasets and compare their performance with that of finetuned LLM models such as T5 and BART. Our results demonstrate that generative mod- els are capable of generating fluent and coher- ent text, achieving BLEU scores of 10.57 and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error analysis re- veals that generative models still struggle with understanding the semantic relations between entities, and they also tend to generate text with hallucinations or irrelevant information. As a part of error analysis, we utilize BERT to de- tect machine-generated text and achieve high macro-F1 scores. We have made the text gener- ated by generative models publicly available.1 1 Introduction Graph-to-text generation is a subtask of data-to- text generation and Natural Language Generation (NLG) (Gatt and Krahmer, 2018). Its purpose is to generate fluent descriptive text based on the structure of a given graph (see Figure 1). With the widespread use of graph structured data, this technique plays a crucial role in various Natural Language Processing applications, including ques- tion answering, dialogue systems, and data aug- mentation (He et al., 2017; Zhao et al., 2020; Josi- foski et al., 2023). Previous research on model architectures has achieved significant performance 1https://github.com/ShuzhouYuan/Eval_ G2T_GenModels on graph-to-text generation benchmarks (Koncel- Kedziorski et al., 2019; Ribeiro et al., 2020; Zhao et al., 2020; Li et al., 2021; Ribeiro et al., 2021b). In particular, Ribeiro et al. (2021a) achieved state- of-the-art performance by employing large pre- trained language models and sufficient training data. However, the zero-shot setting for graph- to-text generation remains challenging due to the inconsistent input format (unstructured text vs. pre- formatted text) between pretraining and fine-tuning stages for large language models. Recently, generative models such as GPT-3 (Brown et al., 2020), InstructGPT (Ouyang et al., 2022), and ChatGPT have gained tremendous atten- tion in both the NLP research community and the general public. Researchers have evaluated these models on various NLP benchmarks in the zero- shot setting (Bang et al., 2023; Jiao et al., 2023; Ahuja et al., 2023). However, their ability to pro- cess structured data, and in particular graph data, such as knowledge graphs, is understudied and worth being explored (Bang et al., 2023). Given the significant resources and annotations required for training graph-to-text generation models (Li et al., 2021), utilizing a zero-shot setting could save training resources and prove advantageous for both economic and ecological reasons. Previous approaches has come up with a neu- ral pipeline to enable zero-shot …

Content truncated to 500 words