PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5531 - 5546 December 7-11, 2022 ©2022 Association for Computational Linguistics PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation Ao Liu1∗, Haoyu Dong2†, Naoaki Okazaki1, Shi Han2, Dongmei Zhang2 1 Tokyo Institute of Technology, 2 Microsoft Research liu.ao@nlp.c.titech.ac.jp, okazaki@c.titech.ac.jp {hadong,shihan,dongmeiz}@microsoft.com Abstract Logical table-to-text generation is a task that involves generating logically faithful sentences from tables, which requires models to derive logical-level facts from table records via logi- cal inference. It raises a new challenge on the logical-level content planning of table-to-text models. However, directly learning the logical inference knowledge from table-text pairs is very difficult for neural models because of the ambiguity of natural language and the scarcity of parallel data. Hence even large-scale pre- trained language models present low logical fidelity on logical table-to-text. In this work, we propose a Pretrained Logical Form Gen- erator (PLOG) framework to improve gener- ation fidelity. Specifically, PLOG is first pre- trained on a table-to-logical-form generation (table-to-logic) task, then finetuned on down- stream table-to-text tasks. The logical forms are formally defined with unambiguous seman- tics. Hence we can collect a large amount of accurate logical forms from tables without hu- man annotation. In addition, PLOG can learn logical inference from table-logic pairs much more reliably than from table-text pairs. To evaluate our model, we further collect a con- trolled logical table-to-text dataset CONTLOG based on an existing dataset. On two bench- marks, LOGICNLG and CONTLOG, PLOG outperforms strong baselines by a large margin on logical fidelity, demonstrating the effective- ness of table-to-logic pretraining. 1 Introduction Table-to-text generation is a sub-task of data-to-text generation, aiming to generate natural language de- scriptions from structured tables. There are two main steps to performing table-to-text generation: content planning (selecting table contents and de- termining the plan to describe them) and surface ∗Work done during internship at MSRA. †Corresponding author. realization (verbalizing the plan into fluent natu- ral language). Traditional table-to-text systems adopt a pipeline architecture to complete the two procedures with separate modules (Kukich, 1983; McKeown, 1985). Recent work has shown the ad- vantage of using a neural encoder-decoder model to directly generate sentences from the tables, which presents the strong capability to produce fluent and natural text (Wiseman et al., 2017; Nie et al., 2018; Puduppully et al., 2019b). Researchers have also attempted to finetune pretrained language models such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2020) on downstream table-to-text tasks and achieved remarkable success on a broad range of benchmarks (Xie et al., 2022; Kale and Rastogi, 2020). Previous studies have mainly focused on surface- level realization, i.e., simply restating surface-level facts in natural language (Wiseman et al., 2017; Liu et al., 2018; Puduppully et al., 2019a,b). Re- cently, logical table-to-text generation (Chen et al., 2020a), i.e., generating textual descriptions that re- quire logical reasoning over surface-level facts in the table, has attracted increasing attention. Logi- cal table-to-text generation poses a new challenge of logical-level content planning, requiring models …

Content truncated to 500 words