How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation?

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7701 - 7710 May 22-27, 2022 c⃝2022 Association for Computational Linguistics How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation? Xunjian Yin and Xiaojun Wan Wangxuan Institute of Computer Technology, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University {xjyin,wanxiaojun}@pku.edu.cn Abstract With the rapid development of deep learn- ing, Seq2Seq paradigm has become prevalent for end-to-end data-to-text generation, and the BLEU scores have been increasing in recent years. However, it is widely recognized that there is still a gap between the quality of the texts generated by models and the texts written by human. In order to better under- stand the ability of Seq2Seq models, evaluate their performance and analyze the results, we choose to use Multidimensional Quality Met- ric(MQM) to evaluate several representative Seq2Seq models on end-to-end data-to-text generation. We annotate the outputs of five models on four datasets with eight error types and find that 1) copy mechanism is helpful for the improvement in Omission and Inaccuracy Extrinsic errors but it increases other types of errors such as Addition; 2) pre-training tech- niques are highly effective, and pre-training strategy and model size are very significant; 3) the structure of the dataset also influences the model’s performance greatly; 4) some spe- cific types of errors are generally challenging for seq2seq models. 1 Introduction Data-to-text generation is a task of automatically producing text from non-linguistic input (Gatt and Krahmer, 2018). The input can be in various forms such as databases of records, spreadsheets, knowl- edge bases, simulations of physical systems. Traditional methods for data-to-text generation (Kukich, 1983; Reiter and Dale, 2000; Mei et al., 2015) implement a pipeline of modules including content planning, sentence planning and surface realization. Recent neural generation systems (Le- bret et al., 2016; Wiseman et al., 2017a) are trained in an end-to-end fashion using the very success- ful encoder-decoder architecture (Bahdanau et al., 2014) as their backbone. Ferreira et al. (2019) intro- duce a systematic and comprehensive comparison between pipeline and end-to-end architectures for this task and conclude that the pipeline models can generate better texts and generalize better to unseen inputs than end-to-end models. However, with the rapid development of the Seq2Seq models especially pre-trained models, more and more end-to-end architectures based on Seq2Seq paradigm get state-of-the-art results on data-to-text benchmarks nowadays. Although BLEU score (Papineni et al., 2002), which is based on precision, has been improved dramatically on standard data-to-text benchmarks such as WebNLG (Gardent et al., 2017), ToTTo (Parikh et al., 2020) and RotoWire (Wiseman et al., 2017b) over the re- cent years, it is commonly accepted that, compared with human evaluation, BLEU score can not evalu- ate the models very well. It is too coarse-grained to reflect the different dimensions of the models’ performance and not always consistent with human judgment (Novikova et al., 2017a; Reiter, 2018; Sulem et al., 2018). Moreover, existing human evaluations on data-to-text generation are usually limited in size …

Content truncated to 500 words