Learning to Generate Textual Data

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1608–1616, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Learning to Generate Textual Data Guillaume Bouchard†‡∗and Pontus Stenetorp†∗ and Sebastian Riedel† {g.bouchard,p.stenetorp,s.riedel}@cs.ucl.ac.uk †Department of Computer Science, University College London ‡Bloomsbury AI Abstract To learn text understanding models with millions of parameters one needs massive amounts of data. In this work, we argue that generating data can compensate for this need. While defining generic data generators is dif- ficult, we propose to allow generators to be “weakly” specified in the sense that a set of parameters controls how the data is generated. Consider for example generators where the ex- ample templates, grammar, and/or vocabulary is determined by this set of parameters. In- stead of manually tuning these parameters, we learn them from the limited training data at our disposal. To achieve this, we derive an ef- ficient algorithm called GENERE that jointly estimates the parameters of the model and the undetermined generation parameters. We il- lustrate its benefits by learning to solve math exam questions using a highly parametrized sequence-to-sequence neural network. 1 Introduction Many tasks require a large amount of training data to be solved efficiently, but acquiring such amounts is costly, both in terms of time and money. In several situations, a human trainer can provide their domain knowledge in the form of a generator of virtual data, such as a negative data sampler for implicit feedback in recommendation systems, physical 3D rendering engines as a simulator of data in a computer vision system, simulators of physical processes to solve science exam question, and math problem genera- tors for automatically solving math word problems. ∗ Contributed equally to this work. Domain-specific data simulators can generate an arbitrary amount of data that can be treated exactly the same way as standard observations, but since they are virtual, they can also be seen as regularizers dedicated to the task we want to solve (Scholkopf and Smola, 2001). While simple, the idea of data simulation is powerful and can lead to significantly better estimations of a predictive model because it prevents overfitting. At the same time it is subject to a strong model bias, because such data genera- tors often generate data that is different from the ob- served data. Creating virtual samples is strongly linked to transfer learning when the task to transfer is corre- lated to the objective (Pan and Yang, 2010). The computer vision literature adopted this idea very early through the notion of virtual samples. Such samples have a natural interpretation: by creating artificial perturbations of an image, its semantics is likely to be unchanged, i.e. training samples can be rotated, blurred, or slightly cropped without chang- ing the category of the objects contained in the im- age (Niyogi et al., 1998). However, for natural language applications the idea of creating invariant transformations is diffi- cult to apply directly, as simple meaning-preserving transformations – such as the replacement of words by their synonyms …

Content truncated to 500 words