Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8199 - 8210 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach Miao Chen§ ∗, Xinjiang Lu¶ �, Tong Xu§, Yanyan Li ¶, Jingbo Zhou¶, Dejing Dou¶, Hui Xiong† ‡ ¶BIL, Baidu Research, §University of Science and Technology of China †Hong Kong University of Science and Technology (Guangzhou) ‡Guangzhou HKUST Fok Ying Tung Research Institute cmer@mail.ustc.edu.cn, {luxinjiang,liyanyanliyanyan,zhoujingbo}@baidu.com, tongxu@ustc.edu.cn, doudejing@baidu.com, xionghui@ust.hk Abstract Although remarkable progress on the neural table-to-text methods has been made, the gen- eralization issues hinder the applicability of these models due to the limited source tables. Large-scale pretrained language models sound like a promising solution to tackle such issues. However, how to effectively bridge the gap between the structured table and the text in- put by fully leveraging table information to fuel the pretrained model is still not well ex- plored. Besides, another challenge of integrat- ing the deliberation mechanism into the text- to-text pretrained model for solving the table- to-text task remains seldom studied. In this paper, to implement the table-to-text genera- tion with pretrained language model, we pro- pose a table structure understanding and text deliberating approach, namely TASD. To be specific, we devise a three-layered multi-head attention network to realize the table-structure- aware text generation model with the help of the pretrained language model. Furthermore, a multi-pass decoder framework is adopted to enhance the capability of polishing generated text for table descriptions. The empirical stud- ies, as well as human evaluation, on two public datasets, validate that our approach can gen- erate faithful and fluent descriptive texts for different types of tables. 1 Introduction The task of learning to generate natural language descriptions from non-linguistic input, which is referred to as data-to-text, is important for many applications, such as weather forecast genera- tion (Mei et al., 2016), sports news writing (Wise- man et al., 2017), biography writing (Lebret et al., 2016), market comments writing (Murakami et al., 2017) and automatic question-answering (Li et al., 2021b). The input data can be in various forms ∗ This work was done when the first author was an intern at Baidu Research under the supervision of the second author. for data-to-text though, here we focus on the text generation task that takes the table as input. Inspired by neural machine translation models, previous studies on table-to-text tasks mainly adopt traditional seq2seq methods to generate table de- scriptions (Lebret et al., 2016; Wiseman et al., 2017; Liu et al., 2018; Gong et al., 2019b; Wang et al., 2020; Li et al., 2021a). Despite generating text with high fluency, lacking numerous source tables leads to lower generalizability of the table- to-text model. Recent progress in the pretrained language model (Devlin et al., 2019; Radford et al., 2019) shows remarkable performance in solving natural language processing tasks. The model pre- trained on large-scale data possesses rich knowl- edge, which inspires us with the potential for …

Content truncated to 500 words