SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1268–1283, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1268 SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness Nathan Ng University of Toronto Vector Institute Kyunghyun Cho New York University Marzyeh Ghassemi University of Toronto Vector Institute Abstract Models that perform well on a training do- main often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a com- mon method used to prevent overfitting and im- prove OOD generalization. However, in natu- ral language, it is difficult to generate new ex- amples that stay on the underlying data man- ifold. We introduce SSMBA, a data augmen- tation method for generating synthetic training examples by using a pair of corruption and re- construction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, lever- aging the manifold assumption to reconstruct corrupted text with masked language mod- els. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA con- sistently outperforms existing data augmenta- tion methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English. 1 1 Introduction Training distributions often do not cover all of the test distributions we would like a supervised clas- sifier or model to perform well on. Often, this is caused by biased dataset collection (Torralba and Efros, 2011) or test distribution drift over time (Quionero-Candela et al., 2009). Therefore, a key challenge in training machine learning models in these settings is ensuring they are robust to unseen examples. Since it is impossible to generalize to the entire distribution, methods often focus on the adjacent goal of out-of-domain robustness. Data augmentation is a common technique used to improve out-of-domain (OOD) robustness by synthetically generating new training examples 1Code is availble at https://github.com/ nng555/ssmba M x x′ ˆx Figure 1: SSMBA moves along the data manifold M by using a corruption function to perturb an example x off the data manifold, then using a reconstruction func- tion to project it back on. (Simard et al., 1998), often by perturbing exist- ing examples in the input space (Perez and Wang, 2017). If data concentrates on a low-dimensional manifold (Chapelle et al., 2006), then these syn- thetic examples should lie in a manifold neigh- borhood of the original examples (Chapelle et al., 2000). Training models to be robust to such lo- cal perturbations has been shown to be effective in improving performance and generalization in semi- supervised and self-supervised settings (Bachman et al., 2014; Szegedy et al., 2014; Sajjadi et al., 2016). When the underlying data manifold exhibits easy-to-characterize properties, as in natural im- ages, simple transformations such as translation and rotation can quickly generate local training examples. However, in domains such as natural language, it is much more difficult to find a set of invariances that preserves meaning or semantics. …

Content truncated to 500 words