Substructure Substitution: Structured Data Augmentation for NLP

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3494–3508 August 1–6, 2021. ©2021 Association for Computational Linguistics 3494 Substructure Substitution: Structured Data Augmentation for NLP Haoyue Shi Karen Livescu Kevin Gimpel Toyota Technological Institute at Chicago, IL, USA, 60637 {freda,klivescu,kgimpel}@ttic.edu Abstract We study a family of data augmentation meth- ods, substructure substitution (SUB2), that gen- eralizes prior methods. SUB2 generates new examples by substituting substructures (e.g., subtrees or subsequences) with others hav- ing the same label. This idea can be ap- plied to many structured NLP tasks such as part-of-speech tagging and parsing. For more general tasks (e.g., text classification) which do not have explicitly annotated substructures, we present variations of SUB2 based on text spans or parse trees, introducing structure- aware data augmentation methods to general NLP tasks. For most cases, training with a dataset augmented by SUB2 achieves bet- ter performance than training with the orig- inal training set. Further experiments show that SUB2 has more consistent performance than other investigated augmentation methods, across different tasks and sizes of the seed dataset.1 1 Introduction Data augmentation has been found effective for various natural language processing (NLP) tasks, such as machine translation (Fadaee et al., 2017; Gao et al., 2019; Xia et al., 2019, inter alia), text classification (Wei and Zou, 2019; Quteineh et al., 2020), syntactic and semantic parsing (Jia and Liang, 2016; Shi et al., 2020; Dehouck and G´omez-Rodr´ıguez, 2020), semantic role labeling (F¨urstenau and Lapata, 2009), and dialogue under- standing (Hou et al., 2018; Niu and Bansal, 2019). Such methods enhance the diversity of the train- ing set by generating examples based on existing ones, and can make the learned models more robust against noise (Xie et al., 2020). Most existing work focuses on word-level manipulation (Kobayashi, 1Project page: https://home.ttic.edu/ ˜freda/project/sub2 2018; Wei and Zou, 2019; Dai and Adel, 2020, inter alia) or global sequence-to-sequence style generation (Sennrich et al., 2016). In this work, we study a family of general data augmentation methods, substructure substitution (SUB2), which generates new examples by substi- tuting same-label substructures (Figure 1). While some instances within this family have been pro- posed before for certain tasks, we generalize the idea and investigate it for a variety of tasks and settings. SUB2 naturally fits structured prediction tasks such as part-of-speech tagging and parsing, where substructures exist in the annotations of the tasks. For more general NLP tasks such as text clas- sification, we present variations of SUB2 which (1) define substructures based on text spans or parse trees for existing examples, and (2) generate new examples by substructure substitution based on the substructures and various kinds of constraints. While data augmentation methods can often be task-specific or have inconsistent performance, ex- tensive experiments show that SUB2 consistently helps models achieve competitive or better perfor- mance than training on the original dataset across structured prediction tasks and original dataset sizes. We further study the effect of different con- straints for the variations of SUB2 in text classifica- tion. While there is …

Content truncated to 500 words