SSMix: Saliency-Based Span Mixup for Text Classification

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3225–3234 August 1–6, 2021. ©2021 Association for Computational Linguistics 3225 SSMix: Saliency-Based Span Mixup for Text Classification Soyoung Yoon1,2∗† Gyuwan Kim1∗ Kyumin Park2 1Clova AI, Naver Corp. 2KAIST {soyoungyoon,pkm9403}@kaist.ac.kr, gyuwan.kim@navercorp.com Abstract Data augmentation with mixup has shown to be effective on various computer vision tasks. Despite its great success, there has been a hurdle to apply mixup to NLP tasks since text consists of discrete tokens with variable length. In this work, we propose SSMix, a novel mixup method where the operation is performed on input text rather than on hidden vectors like previous approaches. SSMix syn- thesizes a sentence while preserving the local- ity of two original texts by span-based mixing and keeping more tokens related to the pre- diction relying on saliency information. With extensive experiments, we empirically vali- date that our method outperforms hidden-level mixup methods on a wide range of text classi- fication benchmarks, including textual entail- ment, sentiment classification, and question- type classification. Our code is available at https://github.com/clovaai/ssmix. 1 Introduction Data augmentation gains popularity in natural lan- guage processing (NLP) (Feng et al., 2021) due to the expensive cost of data collection. Some of them are based on simple rules (Wei and Zou, 2019) and models (Edunov et al., 2018; Ng et al., 2020) to gen- erate similar text. Augmented samples are trained jointly with original samples by a standard way or advanced training methods (Zhu et al., 2019; Park et al., 2021). On the other hand, mixup (Zhang et al., 2018) interpolates input texts and labels for the augmentation. Training with mixup and its variants become a popular regularization method in computer vi- sion to improve the generalization of neural net- works. Mixup approaches are categorized into input-level mixup (Yun et al., 2019; Kim et al., ∗Equal contribution. †Work done during the internship at Clova AI. Figure 1: Illustration of SSMix. Two data samples xA and xB are labeled negative and positive respec- tively for sentiment classification task. For each token, saliency maps are visualized where darker concentra- tion of colors mean higher contribution to correspond- ing label. We select the least salient span from xA and replace it with the most salient span from xB. The out- put results in ˜x = mixup(xA, xB). We also assign ˜y by the mixup ratio λ. In this example, λ is set to 0.2 as the span length is 2 out of 10. 2020; Walawalkar et al., 2020; Uddin et al., 2021) and hidden-level mixup (Verma et al., 2019) de- pending on the location of the mix operation. Input- level mixup is a more prevalent approach than hidden-level mixup because of its simplicity and the ability to capture locality, leading to better ac- curacy. Applying mixup in NLP is more challenging than in computer vision because of the discrete nature of text data and variable sequence lengths. Therefore, most previous attempts on mixup for texts (Guo et al., 2019; Chen et al., 2020) apply mixup …

Content truncated to 500 words