CIAug: Equipping Interpolative Augmentation with Curriculum Learning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1758 - 1764 July 10-15, 2022 ©2022 Association for Computational Linguistics CIAug: Equipping Interpolative Augmentation with Curriculum Learning Ramit Sawhney†∗, Ritesh Soun♣∗, Shrey Pandit§∗, Megh Thakkar§∗, Sarvagya Malaviya⋆, Yuval Pinter△ †Georgia Institute of Technology ♣Sri Venkateswara College, DU §BITS, Pilani ⋆Manipal University Jaipur △Ben-Gurion University of the Negev rsawhney31@gatech.edu, uvp@cs.bgu.ac.il Abstract Interpolative data augmentation has proven to be effective for NLP tasks. Despite its mer- its, the sample selection process in mixup is random, which might make it difficult for the model to generalize better and converge faster. We propose CIAug, a novel curriculum-based learning method that builds upon mixup. It leverages the relative position of samples in hyperbolic embedding space as a complexity measure to gradually mix up increasingly diffi- cult and diverse samples along training. CIAug achieves state-of-the-art results over existing in- terpolative augmentation methods on 10 bench- mark datasets across 4 languages in text clas- sification and named-entity recognition tasks. It also converges and achieves benchmark F1 scores 3 times faster. We empirically analyze the various components of CIAug, and evaluate its robustness against adversarial attacks. 1 Introduction Data augmentation is an effective tool for avoiding overfitting in model training in cases where there is an absence of sufficient training data (Liu et al., 2021). Interpolative augmentation techniques, such as Mixup (Zhang et al., 2018), have shown an in- crease in model performance across various modali- ties, with further improvements gained by applying Mixup at latent representation layers (Chen et al., 2020a). Current implementations of Mixup select samples for interpolation at random, not leverag- ing the potential for adaptive selection techniques which have been shown to lead to better general- izability (Chen et al., 2020b). In addition, these methods do not account for the spatial distribu- tion of linguistic data, known to extend beyond the capacities of euclidean space (Nickel and Kiela, 2017). ∗Equal contribution. Figure 1: Overview of CIAug showing curriculum- based sample selection using hyperbolic distance to perform interpolation. We propose CIAug1, a method which addresses these challenges by offering an augmentation pro- cedure that selects samples in an adaptive fashion and is geometrically sound. CIAug’s sampling strategy follows the idea that selecting easier mix- ing samples first and gradually increasing sample difficulty based on relative spatial position would generate more suitable synthetic inputs, resulting in better model training (Xu et al., 2021). This notion ties in with the framework of curriculum learning (Krueger and Dayan, 2009), where train- ing data is presented in a similarly staggered way, increasing model capabilities (Bengio et al., 2009). CIAug’s selection strategy performs distance oper- ations in hyperbolic space, applying insights about language data spatial distribution to its definition of ‘similar samples’ (Tifrea et al., 2019). We first train CIAug by sampling pairs of sentences similar to each other for mixing, then gradually as training progresses, we sample sentences that are dissimilar, following the curriculum learning strategy. Through experiments …

Content truncated to 500 words