R2D2: Robust Data-to-Text with Replacement Detection

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6903 - 6917 December 7-11, 2022 ©2022 Association for Computational Linguistics R2D2: Robust Data-to-Text with Replacement Detection Linyong Nan Lorenzo Jaime Yu Flores Yilun Zhao Yixin Liu Luke Benson Weijin Zou Yale UniversityDragomir Radev {linyong.nan, lj.flores, yilun.zhao}@yale.edu Abstract Unfaithful text generation is a common prob- lem for text generation systems. In the case of Data-to-Text (D2T) systems, the factuality of the generated text is particularly crucial for any real-world applications. We introduce R2D2, a training framework that addresses unfaithful Data-to-Text generation by training a system both as a generator and a faithfulness discrimi- nator with additional replacement detection and unlikelihood learning tasks. To facilitate such training, we propose two methods for sampling unfaithful sentences. We argue that the poor en- tity retrieval capability of D2T systems is one of the primary sources of unfaithfulness, so in addition to the existing metrics, we further pro- pose named entity based metrics to evaluate the fidelity of D2T generations. Our experimental results show that R2D2 systems could effec- tively mitigate the unfaithful text generation, and they achieve new state-of-the-art results on FeTaQA, LogicNLG, and ToTTo, all with significant improvements. 1 Introduction Data-to-Text generation is the task of generating a text sequence that describes some salient informa- tion of a knowledge source. Unlike Text-to-Text generation whose input source is a text sequence containing knowledge that is not extracted and rep- resented in the canonical structured format, we assume that the input of a Data-to-Text system is represented in some structured format, e.g., RDF (Gardent et al., 2017), relational or entity tables (Lebret et al., 2016; Wiseman et al., 2017). The Data-to-Text task can be divided into two distinct components as in many other text generation tasks (Reiter and Dale, 2000; Gatt and Krahmer, 2018). The first component involves selecting salient infor- mation from the structured knowledge either based on natural language query or other indication of saliency, and the second component comprises or- ganizing and planning of the previous selections to allow realization of the surface text. Although this task has been studied comprehensively in many works, from task design, modeling techniques, to application in different domains (Gardent et al., 2017; Lebret et al., 2016; Wiseman et al., 2017; Novikova et al., 2017; Parikh et al., 2020; Nan et al., 2022), existing Data-to-Text (D2T) systems exhibit a shortcoming that cannot be neglected: they fail to reliably generate sentences that are faithful given the salient content of the input table (Chen et al., 2020a,b, 2021a; Uehara et al., 2020; Ji et al., 2022). This limitation prevents the application of D2T sys- tems in real world scenarios. We therefore need to investigate possible remedies. We introduce a framework that prevents unfaith- ful Data-to-Text generation by training a Data-to- Text system both as a generator as well as a faith- fulness discriminator. For faithfulness discrimina- tion, we adopt the replaced token detection objec- tive, which was first proposed in ELECTRA (Clark et al., 2020). …

Content truncated to 500 words