Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2998–3008 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2998 Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup Guang Liu, Yuzhao Mao, Hailong Huang, Weiguo Gao, and Xuan Li PingAn Life Insurance of China https://github.com/PAI-SmallIsAllYourNeed/Mixup-AMP Abstract Mixup is a recent regularizer for current deep classification networks. Through training a neural network on convex combinations of pairs of examples and their labels, it imposes locally linear constraints on the model’s input space. However, such strict linear constraints often lead to under-fitting which degrades the effects of regularization. Noticeably, this issue is getting more serious when the resource is extremely limited. To address these issues, we propose the Adversarial Mixing Policy (AMP), organized in a “min-max-rand” formulation, to relax the Locally Linear Constraints in Mixup. Specifically, AMP adds a small adversarial per- turbation to the mixing coefficients rather than the examples. Thus, slight non-linearity is in- jected in-between the synthetic examples and synthetic labels. By training on these data, the deep networks are further regularized, and thus achieve a lower predictive error rate. Exper- iments on five text classification benchmarks and five backbone models have empirically shown that our methods reduce the error rate over Mixup variants in a significant margin (up to 31.3%), especially in low-resource condi- tions (up to 17.5%). 1 Introduction Deep classification models have achieved impres- sive results in both images (He et al., 2016; Dosovit- skiy et al., 2020) and language processing (Devlin et al., 2019; Kim, 2014; Wang et al., 2016). One of the most significant challenges to train a deep model is the great efforts and costs to collect large- scale labels. Without sufficient labels, the deep networks tend to generalize poorly, leading to un- satisfactory performance. Thus, the regularization techniques under augmentation schema, which gen- erate labeled data to regularize models (Hernández- García and König, 2018), are widely explored (Wei and Zou, 2019; Liu et al., 2021). Mixup (Zhang et al., 2018) is an effective regu- larizer under the augmentation schema. In recent years, topics related to Mixup have warranted se- rious attention (Lee et al., 2020; Xu et al., 2020; Verma et al., 2019; Archambault et al., 2019; Berth- elot et al., 2019b,a; Beckham et al., 2019; Mao et al., 2019; Zhu et al., 2020). The core idea of Mixup is to generate synthetic training data via a mixing policy, which convex combines a pair of examples and its labels. Through training on these data, the classification networks will be regularized to reach higher performance. Unlike conventional regularizers (Srivastava et al., 2014; Hanson and Pratt, 1988; Ioffe and Szegedy, 2015), Mixup im- poses a kind of locally linear constraint (Zhang et al., 2018; Guo et al., 2019b) on the model’s input space. However, vanilla Mixup often suffers from under- fitting due to the ambiguous data (Guo et al., 2019b; Guo, 2020; Mai et al., 2021) generated under the strict locally linear constraints. To alleviate the under-fitting, (Guo, 2020) …

Content truncated to 500 words