Exploring Story Generation with Multi-task Objectives in Variational Autoencoders

Exploring Story Generation with Multi-task Objectives in Variational Autoencoders Zhuohan Xie Trevor Cohn Jey Han Lau School of Computing and Information Systems The University of Melbourne zhuohanx@student.unimelb.edu.au, t.cohn@unimelb.edu.au, jeyhan.lau@gmail.com Abstract GPT-2 has been frequently adapted in story generation models as it provides powerful gen- erative capability. However, it still fails to generate consistent stories and lacks diversity. Current story generation models leverage ad- ditional information such as plots or common- sense into GPT-2 to guide the generation pro- cess. These approaches focus on improving generation quality of stories while our work look at both quality and diversity. We ex- plore combining BERT and GPT-2 to build a variational autoencoder (VAE), and extend it by adding additional objectives to learn global features such as story topic and discourse re- lations. Our evaluations show our enhanced VAE can provide better quality and diversity trade off, generate less repetitive story content and learn a more informative latent variable. 1 Introduction Autoregressive pretrained models such as GPT-2 (Radford et al., 2019) have been frequently applied to story generation. While GPT-2 can generate co- herent single sentences, it suffers from inconsisten- cies in the storylines and lacks generation diver- sity, i.e. the storylines tend to use “bland” language and multiple generation produces similar plot lines (Guan et al., 2021). Current story generation mod- els add more controllability into language models for story generation, such as story plan (Yao et al., 2019) or commonsense (Guan et al., 2020). These approaches focus on improving generation quality but does not address the diversity issue. Variational autoencoder (VAE) is an extension of autoencoder (AE) (Rumelhart et al., 1986). It defines a prior distribution and the encoder learns an approximate posterior distribution that is opti- mised close to the prior distribution. In doing so, the VAE is able to learn a more tractable latent space than AE and it is easier to sample meaning- ful latent variables to guide the generation process to generate diverse meaningful sequences. In order to leverage pretrained models for VAE, Li et al. (2020) propose OPTIMUS, a large-scale VAE that combines BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019) and further pre- train it on large corpus to create an off-the-shelf pretrained VAE. We follow a similar approach to build our VAE in this paper, but our aim is to de- velop a VAE for domain-specific story generation (rather than creating a domain-general large-scale pretrained VAE) and as such our evaluation focuses on assessing generation capability. Our core innovation in this paper is the introduc- tion of multi-task learning objectives to the VAE to enhance the latent variables, as Bosc and Vincent (2020) found that they tend to learn local features such as the first few words or the length of input sequences. Our first auxiliary objective uses the latent variable to learn story topics, and our sec- ond objective seeks to distinguish between origi- nal stories and “negative samples”, created by alter- ing the stories to simulate common machine gen- eration …

Content truncated to 500 words