ToTTo: A Controlled Table-To-Text Generation Dataset

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1173–1186, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1173 ToTTo: A Controlled Table-To-Text Generation Dataset Ankur P. Parikh♠ Xuezhi Wang♠ Sebastian Gehrmann♠ Manaal Faruqui♠ Bhuwan Dhingra♣∗ Diyi Yang♠♦ Dipanjan Das♠ ♠ Google Research, New York, NY ♦ Georgia Tech, Atlanta, GA ♣ Carnegie Mellon University, Pittsburgh, PA totto@google.com Abstract We present TOTTO, an open-domain English table-to-text dataset with over 120,000 train- ing examples that proposes a controlled gener- ation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing can- didate sentences from Wikipedia. We present systematic analyses of our dataset and anno- tation process as well as results achieved by several state-of-the-art baselines. While usu- ally fluent, existing methods often hallucinate phrases that are not supported by the table, sug- gesting that this dataset can serve as a useful research benchmark for high-precision condi- tional text generation.1 1 Introduction Data-to-text generation (Kukich, 1983; McKeown, 1992) is the task of generating a target textual de- scription y conditioned on source content x in the form of structured data such as a table. Ex- amples include generating sentences given bio- graphical data (Lebret et al., 2016), textual de- scriptions of restaurants given meaning representa- tions (Novikova et al., 2017), basketball game sum- maries given boxscore statistics (Wiseman et al., 2017), and generating fun facts from superlative tables in Wikipedia (Korn et al., 2019). Existing data-to-text tasks have provided an important test-bed for neural generation mod- els (Sutskever et al., 2014; Bahdanau et al., 2014). Neural models are known to be prone to halluci- nation, i.e., generating text that is fluent but not faithful to the source (Vinyals and Le, 2015; Koehn ∗Work done during an internship at Google. 1TOTTO is available at https://github.com/ google-research-datasets/totto. and Knowles, 2017; Lee et al., 2018; Tian et al., 2019) and it is often easier to assess faithfulness of the generated text when the source content is structured (Wiseman et al., 2017; Dhingra et al., 2019). Moreover, structured data can also test a model’s ability for reasoning and numerical infer- ence (Wiseman et al., 2017) and for building repre- sentations of structured objects (Liu et al., 2018), providing an interesting complement to tasks that test these aspects in the NLU setting (Pasupat and Liang, 2015; Chen et al., 2019; Dua et al., 2019). However, constructing a data-to-text dataset can be challenging on two axes: task design and an- notation process. First, tasks with open-ended output like summarization (Mani, 1999; Lebret et al., 2016; Wiseman et al., 2017) lack explicit signals for models on what to generate, which can lead to subjective content and evaluation chal- lenges (Kry´sci´nski et al., 2019). On the other hand, data-to-text tasks that are limited to verbalizing a fully specified meaning representation (Gardent et al., 2017b) do …

Content truncated to 500 words