Neural Text Generation from Structured Data with Application to the Biography Domain

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203–1213, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Neural Text Generation from Structured Data with Application to the Biography Domain R´emi Lebret∗ EPFL, Switzerland David Grangier Facebook AI Research Michael Auli Facebook AI Research Abstract This paper introduces a neural model for concept-to-text generation that scales to large, rich domains. It generates biographical sen- tences from fact tables on a new dataset of biographies from Wikipedia. This set is an order of magnitude larger than existing re- sources with over 700k samples and a 400k vocabulary. Our model builds on conditional neural language models for text generation. To deal with the large vocabulary, we ex- tend these models to mix a fixed vocabulary with copy actions that transfer sample-specific words from the input database to the gener- ated output sentence. To deal with structured data, we allow the model to embed words differently depending on the data fields in which they occur. Our neural model signif- icantly outperforms a Templated Kneser-Ney language model by nearly 15 BLEU. 1 Introduction Concept-to-text generation renders structured records into natural language (Reiter et al., 2000). A typical application is to generate a weather forecast based on a set of structured meteorological mea- surements. In contrast to previous work, we scale to the large and very diverse problem of generating biographies based on Wikipedia infoboxes. An infobox is a fact table describing a person, similar to a person subgraph in a knowledge base (Bollacker et al., 2008; Ferrucci, 2012). Similar generation applications include the generation of product descriptions based on a catalog of millions of items with dozens of attributes each. Previous work experimented with datasets that contain only a few tens of thousands of records such as WEATHERGOV or the ROBOCUP dataset, while our dataset contains over 700k biographies from ∗R´emi performed this work while interning at Facebook. Wikipedia. Furthermore, these datasets have a lim- ited vocabulary of only about 350 words each, com- pared to over 400k words in our dataset. To tackle this problem we introduce a statistical generation model conditioned on a Wikipedia in- fobox. We focus on the generation of the first sen- tence of a biography which requires the model to select among a large number of possible fields to generate an adequate output. Such diversity makes it difficult for classical count-based models to esti- mate probabilities of rare events due to data sparsity. We address this issue by parameterizing words and fields as embeddings, along with a neural language model operating on them (Bengio et al., 2003). This factorization allows us to scale to a larger number of words and fields than Liang et al. (2009), or Kim and Mooney (2010) where the number of parame- ters grows as the product of the number of words and fields. Moreover, our approach does not restrict the re- lations between the field contents and the gener- ated text. This contrasts with less flexible strategies …

Content truncated to 500 words