Logical Natural Language Generation from Open-Domain Tables

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929–7942 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7929 Logical Natural Language Generation from Open-Domain Tables Wenhu Chen1, Jianshu Chen2, Yu Su3, Zhiyu Chen1 and William Yang Wang1 University of California, Santa Barbara, CA, USA1 Tencent AI Lab, Bellevue, WA, USA2 The Ohio State University, Columbus, Ohio, USA3 {wenhuchen, zhiyuchen, william}@cs.ucsb.edu jianshuchen@tencent.com, su.809@osu.edu Abstract Neural natural language generation (NLG) models have recently shown remarkable progress in fluency and coherence. However, existing studies on neural NLG are primarily focused on surface-level realizations with lim- ited emphasis on logical inference, an impor- tant aspect of human thinking and language. In this paper, we suggest a new NLG task where a model is tasked with generating natural lan- guage statements that can be logically entailed by the facts in an open-domain semi-structured table. To facilitate the study of the proposed logical NLG problem, we use the existing Tab- Fact dataset (Chen et al., 2019) featured with a wide range of logical/symbolic inferences as our testbed, and propose new automatic met- rics to evaluate the fidelity of generation mod- els w.r.t. logical inference. The new task poses challenges to the existing monotonic genera- tion frameworks due to the mismatch between sequence order and logical order. In our exper- iments, we comprehensively survey different generation architectures (LSTM, Transformer, Pre-Trained LM) trained with different algo- rithms (RL, Adversarial Training, Coarse-to- Fine) on the dataset and made following obser- vations: 1) Pre-Trained LM can significantly boost both the fluency and logical fidelity met- rics, 2) RL and Adversarial Training are trad- ing fluency for fidelity, 3) Coarse-to-Fine gen- eration can help partially alleviate the fidelity issue while maintaining high language fluency. The code and data are available at https: //github.com/wenhuchen/LogicNLG. 1 Introduction Neural network models, especially the recent wave of massive models like BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), have shown the ability to generate natural language text at an as- tonishing level of fluency and coherence. For the generated text to fulfill its purpose, however, a crit- Nation Gold Medal Silver Medal Bronze Medal Sports Canada 3 1 2 Ice Hockey Mexico 2 3 1 Baseball Colombia 1 3 0 Roller Skating Sentence: Canada obtained 1 more gold medal than Mexico. Sentence: Canada obtained the most gold medals in the game. Medal Table from Tournament Sentence: Canada has got 3 gold medals in the tournament. Sentence: Mexico got 3 silver medals and 1 bronze medal. Surface-level Generation Logical Natural Language Generation Figure 1: Table-to-text generation examples with and without implicit logical inference. Logical NLG re- quires a generation model to generate natural language statements that can be logically entailed by the facts in the table instead of simply restating certain superficial facts in natural language. ical property that is necessary but often overlooked is fidelity, i.e., what is generated should be faith- ful to the underlying data, knowledge, or meaning representation. A line of recent …

Content truncated to 500 words