Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10335 - 10356 December 7-11, 2022 ©2022 Association for Computational Linguistics Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation Zhuang Li△, Lizhen Qu∗ ,△, Qiongkai Xu♡ Tongtong Wu△, Tianyang Zhan† ,♢, Gholamreza Haffari△ Monash University, Australia△,♢, The University of Melbourne, Australia♡ firstname.lastname@monash.edu△, firstname.lastname@unimelb.edu.au♡, tzha225@student.monash.edu♢ Abstract In this paper, we propose a variational au- toencoder with disentanglement priors, VAE- DPRIOR, for task-specific natural language generation with none or a handful of task- specific labeled examples. In order to tackle compositional generalization across tasks, our model performs disentangled representation learning by introducing a conditional prior for the latent content space and another condi- tional prior for the latent label space. Both types of priors satisfy a novel property called ϵ-disentangled. We show both empirically and theoretically that the novel priors can disentan- gle representations even without specific reg- ularizations as in the prior work. The content prior enables directly sampling diverse content representations from the content space learned from the seen tasks, and fuse them with the representations of novel tasks for generating se- mantically diverse texts in the low-resource set- tings. Our extensive experiments demonstrate the superior performance of our model over competitive baselines in terms of i) data aug- mentation in continuous zero/few-shot learning, and ii) text style transfer in the few-shot setting. The code is available at https://github. com/zhuang-li/VAE-DPrior. 1 Introduction Task-specific Natural Language Generation (NLG) aims to generate texts that satisfy desired attributes of target tasks, such as text style transfer (Jin et al., 2020) and task-specific data augmentation (Lee et al., 2021). Herein, a task includes a set of task- specific labels, optionally a set of labeled texts for that task (Han et al., 2020). Although there is al- ready a large amount of labeled data for various tasks, in many application scenarios, such as AI assistants for legal aid, the labeled data of new tasks are still difficult to acquire. As a result, there ∗ corresponding author † Most of this author’s work was finished when he was with Monash University. Figure 1: Generation of task-specific examples for data augmentation. In this example, the content representa- tion is sampled from the training set, while the label representation is constructed based on the support set. may be no or just a handful of labeled texts for target tasks. In such a low-resource setting, given a new task, it is desirable to i) identify which in- formation in texts is task-specific and which is task-independent, and ii) systematically and consis- tently combine the label representations of the new task with task-independent content representations for text generation. As illustrated in Fig. 1, data augmentation needs to combine content represen- tations from seen tasks with novel task labels. In contrast, text style transfer requires combining the content representations extracted from inputs with target styles. Most prior work assumes access to labeled data for supervised training. However, those models trained on seen tasks cannot generalize …

Content truncated to 500 words