An Empirical Survey of Data Augmentation for Limited Data Learning in NLP

An Empirical Survey of Data Augmentation for Limited Data Learning in NLP Jiaao Chen⋄∗ Derek Tam†∗ Colin Raffel† Mohit Bansal† Diyi Yang⋄ ⋄Georgia Institute of Technology, USA †UNC Chapel Hill, USA {jchen896, dyang888}@gatech.edu {dtredsox, craffel, mbansal}@cs.unc.edu Abstract NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets. The dependence on abundant data prevents NLP models from be- ing applied to low-resource settings or novel tasks where significant time, money, or ex- pertise is required to label massive amounts of textual data. Recently, data augmentation methods have been explored as a means of improving data efficiency in NLP. To date, there has been no systematic empiri- cal overview of data augmentation for NLP in the limited labeled data setting, making it difficult to understand which methods work in which settings. In this paper, we pro- vide an empirical survey of recent progress on data augmentation for NLP in the lim- ited labeled data setting, summarizing the landscape of methods (including token-level augmentations, sentence-level augmentations, adversarial augmentations, and hidden-space augmentations) and carrying out experiments on 11 datasets covering topics/news classifica- tion, inference tasks, paraphrasing tasks, and single-sentence tasks. Based on the results, we draw several conclusions to help practitioners choose appropriate augmentations in different settings and discuss the current challenges and future directions for limited data learning in NLP. 1 Introduction Deep learning methods have achieved strong per- formance on a wide range of supervised learn- ing tasks (Sutskever et al., 2014; Deng et al., 2013; Minaee et al., 2021). Traditionally, these re- sults were attained through the use of large, well- labeled datasets. This make them challenging to apply in settings where collecting a large amount of high-quality labeled data for training is expen- sive. Moreover, given the fast-changing nature of real-world applications, it is infeasible to relabel ∗ Equal contribution. every example whenever new data comes in. This highlights a need for learning algorithms that can be trained with a limited amount of labeled data. There has been a substantial amount of research towards learning with limited labeled data for various tasks in the NLP community. One com- mon approach for mitigating the need for labeled data is data augmentation. Data augmentation (Feng et al., 2021) generates new data by modi- fying existing data points through transformations that are designed based on prior knowledge about the problem’s structure (Yang, 2015; Wei and Zou, 2019). This augmented data can be generated from labeled data, and then directly used in super- vised learning (Wei and Zou, 2019), or in semi- supervised learning for unlabeled data through consistencyregularization(Xie et al., 2020)(‘‘con- sistency training’’). While various approaches have been proposed to tackle learning with limited labeled data—including unsupervised pre-training (Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2020), multi-task learning (Glorot et al., 2011; Liu et al., 2017; Augenstein et al., 2018), semi-supervised learning (Zhu, 2005; Chapelle et al., 2009; Miyato et al., 2017; Xie et al., 2020), and few-shot learning (Deng …

Content truncated to 500 words