Faithful Low-Resource Data-to-Text Generation through Cycle Training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2847–2867 July 9-14, 2023 ©2023 Association for Computational Linguistics Faithful Low-Resource Data-to-Text Generation through Cycle Training Zhuoer Wang†1 Marcus Collins⋆2 Nikhita Vedula⋆2 Simone Filice2 Shervin Malmasi2 Oleg Rokhlenko2 1Texas A&M University wang@tamu.edu2Amazon {collmr,veduln,filicesf,malmasi,olegro}@amazon.com Abstract Methods to generate text from structured data have advanced significantly in recent years, pri- marily due to fine-tuning of pre-trained lan- guage models on large datasets. However, such models can fail to produce output faithful to the input data, particularly on out-of-domain data. Sufficient annotated data is often not avail- able for specific domains, leading us to seek an unsupervised approach to improve the faith- fulness of output text. Since the problem is fundamentally one of consistency between the representations of the structured data and text, we evaluate the effectiveness of cycle training in this work. Cycle training uses two models which are inverses of each other: one that gen- erates text from structured data, and one which generates the structured data from natural lan- guage text. We show that cycle training, when initialized with a small amount of supervised data (100 samples in our case), achieves nearly the same performance as fully supervised ap- proaches for the data-to-text generation task on the WebNLG, E2E, WTQ, and WSQL datasets. We perform extensive empirical analysis with automated evaluation metrics and a newly de- signed human evaluation schema to reveal dif- ferent cycle training strategies’ effectiveness of reducing various types of generation errors. Our code is publicly available at https:// github.com/Edillower/CycleNLG. 1 Introduction A wealth of information exists in the form of structured knowledge, such as movie information databases or product catalogs, which we may want to verbalize for a variety of purposes, such as com- paring two items, or presenting detailed descrip- tions in a natural language form suitable for con- versational assistants. Recent work has tackled this data-to-text generation task using freely available †The research was done during an internship at Amazon. *These two authors contributed equally to this work. public datasets, most notably WebNLG (Castro Fer- reira et al., 2020) and ToTTo (Parikh et al., 2020). However, there remain two major challenges. First, the volume of training data required for good per- formance, especially if it is not in a domain rep- resented by one of the existing corpora, is very large. Second, multiple recent papers (Yang et al., 2022; Parikh et al., 2020), inter alia, point out that neural natural language generation (NLG) from structured data tends to produce multiple kinds of errors which limit the utility of these models in customer-facing applications. Hallucinations oc- cur when NLG models inject nonsensical words or information not related to the input structured data, into the generated output text. For instance, an NLG model may claim a shirt’s color is “three”. Simple factual errors occur when an NLG model produces coherent but factually wrong output. There are two threads of research to consider as we attempt to tackle these problems …

Content truncated to 500 words