Latent Predictor Networks for Code Generation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 599–609, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Latent Predictor Networks for Code Generation Wang Ling♦ Edward Grefenstette♦ Karl Moritz Hermann♦ Tom´aˇs Koˇcisk´y♦♣ Andrew Senior♦ Fumin Wang♦ Phil Blunsom♦♣ ♦Google DeepMind ♣University of Oxford {lingwang,etg,kmh,tkocisky,andrewsenior,awaw,pblunsom}@google.com Abstract Many language generation tasks require the production of text conditioned on both structured and unstructured inputs. We present a novel neural network architec- ture which generates an output sequence conditioned on an arbitrary number of in- put functions. Crucially, our approach allows both the choice of conditioning context and the granularity of generation, for example characters or tokens, to be marginalised, thus permitting scalable and effective training. Using this framework, we address the problem of generating pro- gramming code from a mixed natural lan- guage and structured specification. We create two new data sets for this paradigm derived from the collectible trading card games Magic the Gathering and Hearth- stone. On these, and a third preexisting corpus, we demonstrate that marginalis- ing multiple predictors allows our model to outperform strong benchmarks. 1 Introduction The generation of both natural and formal lan- guages often requires models conditioned on di- verse predictors (Koehn et al., 2007; Wong and Mooney, 2006). Most models take the restrictive approach of employing a single predictor, such as a word softmax, to predict all tokens of the output sequence. To illustrate its limitation, suppose we wish to generate the answer to the question “Who wrote The Foundation?” as “The Foundation was written by Isaac Asimov”. The generation of the words “Issac Asimov” and “The Foundation” from a word softmax trained on annotated data is un- likely to succeed as these words are sparse. A ro- bust model might, for example, employ one pre- Figure 1: Example MTG and HS cards. dictor to copy “The Foundation” from the input, and a another one to find the answer “Issac Asi- mov” by searching through a database. However, training multiple predictors is in itself a challeng- ing task, as no annotation exists regarding the pre- dictor used to generate each output token. Fur- thermore, predictors generate segments of differ- ent granularity, as database queries can generate multiple tokens while a word softmax generates a single token. In this work we introduce Latent Predictor Networks (LPNs), a novel neural archi- tecture that fulfills these desiderata: at the core of the architecture is the exact computation of the marginal likelihood over latent predictors and gen- erated segments allowing for scalable training. We introduce a new corpus for the automatic generation of code for cards in Trading Card Games (TCGs), on which we validate our model 1. TCGs, such as Magic the Gathering (MTG) and Hearthstone (HS), are games played between two players that build decks from an ever expanding pool of cards. Examples of such cards are shown in Figure 1. Each card is identified by its attributes 1Dataset available at https://deepmind.com/publications.html 599 (e.g., name and cost) and has an …

Content truncated to 500 words