Challenges in Data-to-Document Generation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253–2263 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Challenges in Data-to-Document Generation Sam Wiseman and Stuart M. Shieber and Alexander M. Rush School of Engineering and Applied Sciences Harvard University Cambridge, MA, USA {swiseman,shieber,srush}@seas.harvard.edu Abstract Recent neural models have shown signif- icant progress on the problem of generat- ing short descriptive texts conditioned on a small number of database records. In this work, we suggest a slightly more dif- ficult data-to-text generation task, and in- vestigate how effective current approaches are on this task. In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, pro- pose a series of extractive evaluation meth- ods for analyzing performance, and ob- tain baseline results using current neural generation methods. Experiments show that these models produce fluent text, but fail to convincingly approximate human- generated documents. Moreover, even templated baselines exceed the perfor- mance of these neural models on some metrics, though copy- and reconstruction- based extensions lead to noticeable im- provements. 1 Introduction Over the past several years, neural text genera- tion systems have shown impressive performance on tasks such as machine translation and summa- rization. As neural systems begin to move toward generating longer outputs in response to longer and more complicated inputs, however, the gener- ated texts begin to display reference errors, inter- sentence incoherence, and a lack of fidelity to the source material. The goal of this paper is to suggest a particular, long-form generation task in which these challenges may be fruitfully explored, to provide a publically available dataset for this task, to suggest some automatic evaluation met- rics, and finally to establish how current, neural text generation methods perform on this task. A classic problem in natural-language genera- tion (NLG) (Kukich, 1983; McKeown, 1992; Re- iter and Dale, 1997) involves taking structured data, such as a table, as input, and producing text that adequately and fluently describes this data as output. Unlike machine translation, which aims for a complete transduction of the sentence to be translated, this form of NLG is typically taken to require addressing (at least) two separate chal- lenges: what to say, the selection of an appropriate subset of the input data to discuss, and how to say it, the surface realization of a generation (Reiter and Dale, 1997; Jurafsky and Martin, 2014). Tra- ditionally, these two challenges have been modu- larized and handled separately by generation sys- tems. However, neural generation systems, which are typically trained end-to-end as conditional lan- guage models (Mikolov et al., 2010; Sutskever et al., 2011, 2014), blur this distinction. In this context, we believe the problem of generating multi-sentence summaries of tables or database records to be a reasonable next-problem for neural techniques to tackle as they begin to consider more difficult NLG tasks. In particu- lar, we would like this generation task to have the following two properties: (1) it is relatively easy to obtain fairly clean …

Content truncated to 500 words