Logic-Consistency Text Generation from Semantic Parses

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4414–4426 August 1–6, 2021. ©2021 Association for Computational Linguistics 4414 Logic-Consistency Text Generation from Semantic Parses Chang Shu1*, Yusen Zhang2*, Xiangyu Dong3, Peng Shi4, Tao Yu5, Rui Zhang2 1School of Informatics, University of Edinburgh 2Department of Computer Science and Engineering, Penn State University 3School of Computer Science and Engineering, Beihang University 4David R. Cheriton School of Computer Science, University of Waterloo 5Department of Computer Science, The University of Hong Kong s1783039@ed.ac.uk, yfz5488@psu.edu, dxy912434027@gmail.com peng.shi@uwaterloo.ca, rmz5227@psu.edu Abstract Text generation from semantic parses is to generate textual descriptions for formal rep- resentation inputs such as logic forms and SQL queries. This is challenging due to two reasons: (1) the complex and intensive in- ner logic with the data scarcity constraint, (2) the lack of automatic evaluation metrics for logic consistency. To address these two chal- lenges, this paper first proposes SNOWBALL, a framework for logic consistent text genera- tion from semantic parses that employs an it- erative training procedure by recursively aug- menting the training set with quality control. Second, we propose a novel automatic met- ric, BLEC, for evaluating the logical consis- tency between the semantic parses and gener- ated texts. The experimental results on two benchmark datasets, Logic2Text and Spider, demonstrate the SNOWBALL framework en- hances the logic consistency on both BLEC and human evaluation. Furthermore, our sta- tistical analysis reveals that BLEC is more log- ically consistent with human evaluation than general-purpose automatic metrics including BLEU, ROUGE and, BLEURT. Our data and code are available at https://github.com/ Ciaranshu/relogic. 1 Introduction Natural language generation (NLG) from semantic parses is to generate the text description for the formal representation input such as logical forms, AMR, and SQL queries. It has drawn widespread attention because of its substantial contributions to the interpretability and usability of the latest natural language interfaces (Gatt and Krahmer, 2018; Chen et al., 2020b; Hu et al., 2020; Mishra et al., 2019; Yu et al., 2019; Ngomo et al., 2013; Wang et al., 2018; Gardent et al., 2017; Wang et al., 2020a; *Equal Contribution Figure 1: Our data augmentation procedure for the gen- erator and evaluator in the SNOWBALL framework. Wang, 2019; Koutrika et al., 2010a). Recently, pre- trained large-scale language models like BERT (De- vlin et al., 2018), T5 (Raffel et al., 2020), and GPT- 3 (Brown et al., 2020) have raised the ability to generate natural language from formal texts to a promising level of fluency and coherence. However, NLG from semantic parses still has suffered from two crucial challenges: (1) the data scarcity constraint due to the bias on certain types of logic forms or expensive labeling work (Iyer et al., 2017; Yaghmazadeh et al., 2017), which po- tentially leads to the unsatisfied fidelity of remain- ing the complex and intensive inner logic in the generated text based on our empirical research; (2) The general-purpose automatic metrics (Novikova et al., 2017a) such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and BLEURT (Sellam et …

Content truncated to 500 words