Controllable Text Generation with Focused Variation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 351–366 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 351 Control, Generate, Augment: A Scalable Framework for Multi-Attribute Text Generation Giuseppe Russo1, Nora Hollenstein1, Claudiu Musat2, Ce Zhang1 1 ETH Zurich, {russog,noraho,ce.zhang}@inf.ethz.ch 2 Swisscom, claudiu.musat@swisscom.com Abstract We introduce CGA, a conditional VAE ar- chitecture, to control, generate, and augment text. CGA is able to generate natural En- glish sentences controlling multiple semantic and syntactic attributes by combining adver- sarial learning with a context-aware loss and a cyclical word dropout routine. We demon- strate the value of the individual model com- ponents in an ablation study. The scalabil- ity of our approach is ensured through a sin- gle discriminator, independently of the num- ber of attributes. We show high quality, di- versity and attribute control in the generated sentences through a series of automatic and human assessments. As the main application of our work, we test the potential of this new NLG model in a data augmentation scenario. In a downstream NLP task, the sentences gen- erated by our CGA model show significant improvements over a strong baseline, and a classification performance often comparable to adding same amount of additional real data. 1 Introduction Recently, natural language generation (NLG) has become a prominent research topic in NLP due to its diverse applications, ranging from machine translation (e.g., Sennrich et al. (2016)) to dialogue systems (e.g., Budzianowski and Vuli´c (2019)). The common goal of these applications using au- tomatic text generation is the augmentation of datasets used for supervised NLP tasks. To this end, one of the key demands of NLG is controlled text generation, more specifically, the ability to sys- tematically control semantic and syntactic aspects of generated text. Most previous approaches simplify this problem by approximating NLG with the control of one single attribute of the text, such as sentiment or formality (e.g., Li et al. (2018), Fu et al. (2018), and John et al. (2019)). However, the problem of controlled generation typically relies on multiple components such as lexical, syntactic, semantic and stylistic aspects. Therefore, the simultaneous con- trol of multiple attributes becomes vital to generate natural sentences suitable for downstream tasks. Methods such as the ones presented by Hu et al. (2017) and Subramanian et al. (2018) succeed in simultaneously controlling various attributes. How- ever, these methods depend on the transformation of input reference sentences, or do not scale eas- ily to more than two attributes due to architectural complexities, such as the requirement for separate discriminators for each additional attribute. In light of these challenges, we propose the Control, Generate, Augment framework (CGA), a powerful model to synthesize additional labeled data sampled from a latent space. The accurate multi-attribute control of our approach offers sig- nificant performance gains on downstream NLP tasks. We provide the code and all generated En- glish sentences to facilitate future research1. The main contribution of this paper is a scalable model which learns to control multiple semantic and syntactic attributes …

Content truncated to 500 words