A Survey of Data Augmentation Approaches for NLP

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988 August 1–6, 2021. ©2021 Association for Computational Linguistics 968 A Survey of Data Augmentation Approaches for NLP Steven Y. Feng∗, 1 Varun Gangal∗, 1 Jason Wei†, 2 Sarath Chandar,3 Soroush Vosoughi,4 Teruko Mitamura,1 Eduard Hovy1 1Carnegie Mellon University, 2Google Research 3Mila - Quebec AI Institute, 4Dartmouth College {syfeng,vgangal,teruko,hovy}@cs.cmu.edu jasonwei@google.com sarath.chandar@mila.quebec soroush@dartmouth.edu Abstract Data augmentation has recently seen increased interest in NLP due to more work in low- resource domains, new tasks, and the popu- larity of large-scale neural networks that re- quire large amounts of training data. De- spite this recent upsurge, this area is still rel- atively underexplored, perhaps due to the chal- lenges posed by the discrete nature of language data. In this paper, we present a comprehen- sive and unifying survey of data augmenta- tion for NLP by summarizing the literature in a structured manner. We first introduce and motivate data augmentation for NLP, and then discuss major methodologically representative approaches. Next, we highlight techniques that are used for popular NLP applications and tasks. We conclude by outlining current chal- lenges and directions for future research. Over- all, our paper aims to clarify the landscape of existing literature in data augmentation for NLP and motivate additional work in this area. We also present a GitHub repository with a pa- per list that will be continuously updated at https://github.com/styfeng/DataAug4NLP. 1 Introduction Data augmentation (DA) refers to strategies for in- creasing the diversity of training examples without explicitly collecting new data. It has received active attention in recent machine learning (ML) research in the form of well-received, general-purpose tech- niques such as UDA (Xie et al., 2020) (3.1) and MIXUP (Zhang et al., 2017) (3.2). These are often first explored in computer vision (CV), and DA’s adaptation for natural language processing (NLP) seems secondary and comparatively underexplored, perhaps due to challenges presented by the discrete nature of language, which rules out continuous noising and makes it hard to maintain invariance. ∗ Equal contribution by the two authors. † AI Resident. Figure 1: Weekly Google Trends scores for the search term "data augmentation", with a control, uneventful ML search term ("minibatch") for comparison. Despite these challenges, there has been in- creased interest and demand for DA for NLP. As NLP grows due to off-the-shelf availability of large pretrained models, there are increasingly more tasks and domains to explore. Many of these are low-resource, and have a paucity of training exam- ples, creating many use-cases for which DA can play an important role. Particularly, for many non- classification NLP tasks such as span-based tasks and generation, DA research is relatively sparse despite their ubiquity in real-world settings. Our paper aims to sensitize the NLP community towards this growing area of work, which has also seen increasing interest in ML overall (as seen in Figure 1). As interest and work on this topic con- tinue to increase, this is an opportune time for a paper of our kind to (i) give …

Content truncated to 500 words