TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5243 - 5258 July 10-15, 2022 ©2022 Association for Computational Linguistics TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding Le Zhang Fudan University zhangle18@fudan.edu.cn Zichao Yang CMU yangtze2301@gmail.com Diyi Yang Georgia Tech dyang888@gatech.edu Abstract Data augmentation is an effective approach to tackle over-fitting. Many previous works have proposed different data augmentations strate- gies for NLP, such as noise injection, word replacement, back-translation etc. Though ef- fective, they missed one important characteris- tic of language–compositionality, meaning of a complex expression is built from its sub-parts. Motivated by this, we propose a compositional data augmentation approach for natural lan- guage understanding called TreeMix. Specifi- cally, TreeMix leverages constituency parsing tree to decompose sentences into constituent sub-structures and the Mixup data augmenta- tion technique to recombine them to generate new sentences. Compared with previous ap- proaches, TreeMix introduces greater diversity to the samples generated and encourages mod- els to learn compositionality of NLP data. Ex- tensive experiments on text classification and SCAN demonstrate that TreeMix outperforms current state-of-the-art data augmentation meth- ods. 1 Introduction Data augmentation (DA) has won great popular- ity in natural language processing (NLP) (Chen et al., 2021; Feng et al., 2021) due to the increas- ing demand for data and the expensive cost for annotation. DA aims at increasing the quantity and diversity of the datasets by generating more samples based on existing ones, which helps make the training process more consistent and improves the model’s capacity for generalization (Xie et al., 2020). For instance, existing DA methods often leverage word-level manipulation (Wei and Zou, 2019; Kobayashi, 2018; Karimi et al., 2021) and model-based sentence generation (Edunov et al., 2018; Ng et al., 2020). As mixup-based (Zhang et al., 2018) augmentation achieving huge success in computer vision (Yun et al., 2019; Uddin et al., 2021; Kim et al., 2021), some recent works start S1:They will find little interest in this poor film. S2:It comes as a touching love story. Method Example EDA (Wei and Zou, 2019) They will this find little interest in bad movie. AEDA (Karimi et al., 2021) They will find ? little in ! this poor movie;. Noise (Xie et al., 2017) Thes will fi little intres _ this poor film . SSMix (Yoon et al., 2021) They will find little interest in love poor film Replacement (Kolomiyets et al., 2011) They will find limited interest in this odd film. Back Translation (Edunov et al., 2018) They will show little interest in this strange film. TreeMix They will find little interest in this touching love story. Table 1: Input-level DAs for Text-Classification. EDA includes random deletion, swapping, and insertion. AEDA randomly inserts punctuation. SSMix swaps tokens based on their saliency. The replacement method randomly substitutes words with synonyms. In Back- translation, the source sentences are first translated into another language, and then back again. to adapt mixup to NLP, such as at the hidden level …

Content truncated to 500 words