Generating Sentences by Editing Prototypes

Generating Sentences by Editing Prototypes Kelvin Guu*2 Tatsunori B. Hashimoto*1,2 Yonatan Oren1 Percy Liang1,2 (* equal contribution) 1Department of Computer Science 2Department of Statistics Stanford University {kguu,thashim,yonatano}@stanford.edu pliang@cs.stanford.edu Abstract We propose a new generative language model for sentences that first samples a prototype sentence from the training corpus and then ed- its it into a new sentence. Compared to tra- ditional language models that generate from scratch either left-to-right or by first sampling a latent sentence vector, our prototype-then- edit model improves perplexity on language modeling and generates higher quality outputs according to human evaluation. Furthermore, the model gives rise to a latent edit vector that captures interpretable semantics such as sen- tence similarity and sentence-level analogies. 1 Introduction The ability to generate sentences is core to many NLP tasks, including machine translation, summa- rization, speech recognition, and dialogue. Most neural models for these tasks are based on recur- rent neural language models (NLMs), which gener- ate sentences from scratch, often in a left-to-right manner (Bengio et al., 2003). It is often observed that such NLMs suffer from the problem of favoring generic utterances such as “I don’t know” (Li et al., 2016). At the same time, naive strategies to increase diversity have been shown to compromise grammat- icality (Shao et al., 2017), suggesting that current NLMs may lack the inductive bias to faithfully rep- resent the full diversity of complex utterances. Indeed, it is difficult even for humans to write complex text from scratch in a single pass; we of- ten create an initial draft and incrementally revise it (Hayes and Flower, 1986). Inspired by this process, Overpriced , overrated , and tasteless food . The food here is ok but not worth the price . I definitely recommend this restaurante . Sample from the training set Edit using a�en�on Edit Vector The food here is ok but not worth the price . Prototype The food is mediocre and not worth the ridiculous price . Generation Generation The food is good but not worth the horrible customer service . The food here is not worth the drama . The food is not worth the price . Figure 1: The prototype-then-edit model generates a sen- tence by sampling a random example from the training set and then editing it using a randomly sampled edit vector. we propose a new unconditional generative model of text which we call the prototype-then-edit model, illustrated in Figure 1. It first samples a random pro- totype sentence from the training corpus, and then invokes a neural editor, which draws a random “edit vector” and generates a new sentence by attending to the prototype while conditioning on the edit vec- tor. The motivation is that sentences from the corpus provide a high quality starting point: they are gram- matical, naturally diverse, and exhibit no bias to- wards shortness or vagueness. The attention mech- anism (Bahdanau et al., 2015) of the neural editor strongly biases the generation towards the prototype, and therefore it needs to solve a …

Content truncated to 500 words