Dynamic Nonlinear Mixup with Distance-based Sample Selection

Proceedings of the 29th International Conference on Computational Linguistics, pages 3788–3797 October 12–17, 2022. 3788 Dynamic Nonlinear Mixup with Distance-based Sample Selection Shaokang Zhang1,2, Lei Jiang1,2 ∗, Jianlong Tan1,2 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China zhangshaokang@iie.ac.cn; jianglei@iie.ac.cn; tanjianlong@iie.ac.cn Abstract Data augmentation with mixup has shown to be effective on the NLP tasks. Although its great success, the mixup still has shortcomings. First, vanilla mixup randomly selects one sam- ple to generate the mixup sample for a given sample. It remains unclear how to best choose the input samples for the mixup. Second, lin- ear interpolation limits the space of synthetic data and its regularization effect. In this pa- per, we propose the dynamic nonlinear mixup with distance-based sample selection, which not only generates multiple sample pairs based on the distance between each sample but also enlarges the space of synthetic samples. Specif- ically, we compute the distance between each input data by cosine similarity and select multi- ple samples for a given sample. Then we use the dynamic nonlinear mixup to fuse sample pairs. It does not use a linear, scalar mixing strategy, but a nonlinear interpolation strategy, where the mixing strategy is adaptively updated for the input and label pairs. Experiments on the multiple public datasets demonstrate that dynamic nonlinear mixup outperforms state-of- the-art methods. 1 Introduction Deep neural networks have achieved great success in NLP tasks, such as text classification (Zhang et al., 2015), machine translation (Sutskever et al., 2014), and dialogue tasks (Serban et al., 2016). These models usually require a large amount of la- beled data, but labeling data is time-consuming and expensive. Data augmentation is a common tech- nology to solve the data scarcity problem. Some of them are based on rules (Wei and Zou, 2019) and models (Edunov et al., 2018) to generate similar text. Augmented data are trained by advanced train- ing methods (Park et al., 2021). On the other hand, mixup (Zhang et al., 2017b) trains the classifier on ∗Corresponding author the synthetic data which are generated by the linear interpolation of the input and label pairs. Recently, many variants of mixup are proposed and successfully applied to NLP tasks. It mainly includes two categories: input-level mixup (Yoon et al., 2021) and hidden-level mixup (Sun et al., 2020; Guo, 2020). Hidden-level mixup is more popular because of the discrete nature of text data and variable sequence lengths. It usually fuses the hidden vectors like embeddings or intermediate representations. While the hidden-level mixup is effective, it still has some issues. First, mixup produces the sample pairs by randomly choosing a sample for a given sample. How to select the optimal input and label pairs is unclear. Second, the space of synthetic data is limited due to its linear nature. Although nonlinear mixup (Guo, 2020) is proposed and utilizes the matrix mixing policy for the sample pairs, the matrix is fixed during the training process and it is …

Content truncated to 500 words