Leveraging QA Datasets to Improve Generative Data Augmentation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9737 - 9750 December 7-11, 2022 ©2022 Association for Computational Linguistics Leveraging QA Datasets to Improve Generative Data Augmentation Dheeraj Mekala♢ Tu Vu♠ Timo Schick♣ Jingbo Shang♢,♡,∗ ♢ University of California San Diego ♠ University of Massachusetts Amherst ♣ Meta AI Research ♡ Halıcıo˘glu Data Science Institute, University of California San Diego ♢ {dmekala, jshang}@ucsd.edu ♠ tuvu@cs.umass.edu ♣ schick@fb.com Abstract The ability of generative language models (GLMs) to generate text has improved consid- erably in the last few years, enabling their use for generative data augmentation. In this work, we propose CONDA, an approach to further im- prove GLMs’ ability to generate synthetic data by reformulating data generation as context generation for a given question-answer (QA) pair and leveraging QA datasets for training context generators. Then, we cast downstream tasks into the same question answering format and adapt the fine-tuned context generators to the target task domain. Finally, we use the fine-tuned GLM to generate relevant contexts, which are in turn used as synthetic training data for their corresponding tasks. We perform ex- tensive experiments on multiple classification datasets and demonstrate substantial improve- ments in performance for both few- and zero- shot settings. Our analysis reveals that QA datasets that require high-level reasoning abil- ities (e.g., abstractive and common-sense QA datasets) tend to give the best boost in perfor- mance in both few-shot and zero-shot settings. 1 Introduction Recent advances in NLP have substantially im- proved the capability of pretrained language mod- els to generate high-quality text (Radford and Narasimhan, 2018; Radford et al., 2019; Lewis et al., 2020; Brown et al., 2020). Various ap- proaches (e.g., Kumar et al., 2020; Anaby-Tavor et al., 2020; Mekala et al., 2021) leverage this ca- pability for generative data augmentation. This process usually involves first fine-tuning the GLM on training samples prepended with their target label and then generating synthetic data by prompt- ing the GLM with a given target label. However, it is not evident that the model parameters learnt during pretraining or fine-tuning should support ∗ Jingbo Shang is the corresponding author. Text Label (topic/sentiment) hockey is a great game sports this is the worst movie negative Question Answer Context what is the document about? sports hockey is a great game Is the movie good or bad? negative this is the worst movie. General text classification format Question-Answer-Context format Figure 1: Examples of converting topic classifica- tion and sentiment analysis data into question-answer- context format. data generation using such unintuitive formulations with label tokens as prompts: In low data regimes, fine-tuning can be unstable (Devlin et al., 2019) and relies on the pretrained parameters to be rea- sonably well-suited for the target task (Phang et al., 2018). Therefore, for target domains that are differ- ent from the pretraining domain, such formulations may result in poor quality generation (Feng et al., 2020). To address this challenge, we propose CONDA, an approach to leverage existing QA datasets for training Context …

Content truncated to 500 words