Structure-Aware Pre-Training for Table-to-Text Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2273–2278 August 1–6, 2021. ©2021 Association for Computational Linguistics 2273 Structure-Aware Pre-Training for Table-to-Text Generation Xinyu Xing and Xiaojun Wan Wangxuan Institute of Computer Technology, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University {xingxinyu,wanxiaojun}@pku.edu.cn Abstract Table-to-text generation is a subtask of data- to-text generation which aims to generate nal- tural language text based on input table. Pre- training techniques have achieved great suc- cess on table-to-text generation. However, the pre-trained models used in previous works are typically trained on free-form natural language text while the input of table-to-text task is structured table. In this paper, we propose STTP, a pre-trained model that is trained with tables and their contexts. The STTP model can understand the structured input table and gen- erate fluent text. Experiments on two datasets show the efficacy of our model. 1 Introduction Data-to-text generation (Reiter and Dale, 1997) is an important natural language generation task with many practical applications, and it refers to the task of generating textual output from non- linguistic input data. The input data of the task can include tables of records, simulations of phys- ical systems, spreadsheets, and so on. The output of the task is a natural language text. Datasets in common use include WEATHERGOV(Liang et al., 2009), ROTOWIRE(Wiseman et al., 2017), WebNLG(Gardent et al., 2017) and so on. Neu- ral generation models with different improvements have achieved impressive results on data-to-text task. Table-to-text generation is a subtask of data- to-text generation which takes tables as input. The pretrain-and-finetune framework, which refers to first pre-training a high capacity model on large corpora and then fine-tuning it on a down- stream task, has outperformed prior state of the art on both natural language understanding task and natural language generation task. Inspired by the success of transfer learning, recently some works (Mager et al., 2020; Kale, 2020; Ribeiro et al., 2020) try to apply the pretrain-and-finetune framework on data-to-text generation. They fine- tuned the pre-trained model such as BART(Lewis et al., 2019) or T5(Raffel et al., 2019) on several downstream data-to-text tasks and achieved state- of-the-art results. Although transfer learning has achieved great success on data-to-text generation, the pre-trained models used in previous works are typically trained on free-form natural language texts while the input of table-to-text task is structured table. The text- to-text pre-trained models learn a lot of knowledge and good language models from large amount of texts, so they work well on data-to-text generation task. But they still lack the ability to understand the structured data. So we propose a structure- aware table-to-text pre-trained model STTP which is trained with tables and their contexts for table- to-text task. STTP is built on top of the text-to-text pre-trained model BART, and it can understand the structured table and describe it with natural lan- guage text. We train our model based on BART because we hope our model can benefit from the knowledge and language model learned from large corpora. …

Content truncated to 500 words