When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation

Findings of the Association for Computational Linguistics: ACL 2022, pages 1048 - 1062 May 22-27, 2022 c⃝2022 Association for Computational Linguistics When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation Ehsan Kamalloo∗ † Univeristy of Alberta kamalloo@ualberta.ca Mehdi Rezagholizadeh∗ Huawei Noah’s Ark Lab mehdi.rezagholizadeh@huawei.com Ali Ghodsi University of Waterloo ali.ghodsi@uwaterloo.ca Abstract Data Augmentation (DA) is known to improve the generalizability of deep neural networks. Most existing DA techniques naively add a certain number of augmented samples without considering the quality and the added compu- tational cost of these samples. To tackle this problem, a common strategy, adopted by sev- eral state-of-the-art DA methods, is to adap- tively generate or re-weight augmented sam- ples with respect to the task objective dur- ing training. However, these adaptive DA methods: (1) are computationally expensive and not sample-efficient, and (2) are designed merely for a specific setting. In this work, we present a universal DA technique, called Glit- ter, to overcome both issues. Glitter can be plugged into any DA method, making train- ing sample-efficient without sacrificing perfor- mance. From a pre-generated pool of aug- mented samples, Glitter adaptively selects a subset of worst-case samples with maximal loss, analogous to adversarial DA. Without altering the training strategy, the task objec- tive can be optimized on the selected sub- set. Our thorough experiments on the GLUE benchmark, SQuAD, and HellaSwag in three widely used training setups including consis- tency training, self-distillation and knowledge distillation reveal that Glitter is substantially faster to train and achieves a competitive per- formance, compared to strong baselines.1 1 Introduction The undeniable importance of data in deep learn- ing (Sambasivan et al., 2021; Rogers, 2021) and the costly process of data annotation has propelled re- searchers into leveraging Data Augmentation (DA) in a broad range of applications from computer vision (Cubuk et al., 2019; Wang et al., 2020) to ∗Equal Contribution. †Work done while interning at Huawei Noah’s Ark Lab. 1Our code is available at https://github.com/ huawei-noah/KD-NLP/tree/main/Glitter. natural language processing (NLP) including ma- chine translation (Sennrich et al., 2016; Shen et al., 2020), language understanding (Shen et al., 2020; Qu et al., 2021; Du et al., 2021; Kamalloo et al., 2021), and question answering (Alberti et al., 2019; Longpre et al., 2019; Shakeri et al., 2020). DA is shown to be effective in improving generaliza- tion of deep neural networks (DeVries and Taylor, 2017; Xie et al., 2020) and in increasing the num- ber of training samples especially in low resource data regimes (Sennrich et al., 2016; Zhang et al., 2018). Nonetheless, in NLP, the discrete nature of text poses additional complexity to DA as gener- ating semantically viable text from another text is challenging (Feng et al., 2021). DA methods can be broadly categorized into task-aware and task-agnostic methods. Task- agnostic DA methods essentially generate aug- mented text regardless of the task at hand and often do not warrant additional training or fine-tuning. They can be based on some hand-crafted heuristics (Zhang et …

Content truncated to 500 words