Investigating the Robustness of Natural Language Generation from Logical Forms via Counterfactual Samples

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5499 - 5512 December 7-11, 2022 ©2022 Association for Computational Linguistics Investigating the Robustness of Natural Language Generation from Logical Forms via Counterfactual Samples Chengyuan Liu1∗, Leilei Gan2∗†, Kun Kuang2†, Fei Wu2,3,4 1School of Software Technology, Zhejiang University 2College of Computer Science and Technology, Zhejiang University 3Shanghai Institute for Advanced Study of Zhejiang University 4Shanghai AI Laboratory {liucy1, leileigan, kunkuang, wufei}@zju.edu.cn Abstract The aim of Logic2Text is to generate control- lable and faithful texts conditioned on tables and logical forms, which not only requires a deep understanding of the tables and logical forms, but also warrants symbolic reasoning over the tables. State-of-the-art methods based on pre-trained models have achieved remark- able performance on the standard test dataset. However, we question whether these methods really learn how to perform logical reasoning, rather than just relying on the spurious corre- lations between the headers of the tables and operators of the logical form. To verify this hy- pothesis, we manually construct a set of coun- terfactual samples, which modify the original logical forms to generate counterfactual logical forms with rarely co-occurred table headers and logical operators. SOTA methods give much worse results on these counterfactual samples compared with the results on the original test dataset, which verifies our hypothesis. To deal with this problem, we firstly analyze this bias from a causal perspective, based on which we propose two approaches to reduce the model’s reliance on the shortcut. The first one incor- porates the hierarchical structure of the logical forms into the model. The second one exploits automatically generated counterfactual data for training. Automatic and manual experimen- tal results on the original test dataset and the counterfactual dataset show that our method is effective to alleviate the spurious correlation. Our work points out the weakness of previous methods and takes a further step toward de- veloping Logic2Text models with real logical reasoning ability. 1 Introduction Recently, generating logical consistent natural language from tables has attracted the atten- tion of the research community (Wiseman et al., *Equal contribution. †Corresponding author. Figure 1: The GPT-2 based model makes a fluent and logical consistent prediction on the original Logic2Text sample, but generates a sentence logically inconsistent with the logical form of the counterfactual sample. 2018; Lee, 2018; Liang et al., 2009; Shu et al., 2021; Chen et al., 2020c,b). Given a table, the Logic2Text (Chen et al., 2020d) task is required to generate controllable and faithful texts conditioned on a logical form, which not only requires a deep understanding of the tables and logical forms, but also warrants symbolic reasoning over the tables. 5499 State-of-the-art methods based on pre-trained mod- els (Radford et al., 2019; Shu et al., 2021) have achieved remarkable performance on the standard test dataset of Logic2Text. Figure 1 shows an orig- inal sample of the task. However, we question whether these methods re- ally learn how to perform logical reasoning, rather than just relying on the spurious correlations …

Content truncated to 500 words