HypMix: Hyperbolic Interpolative Data Augmentation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9858–9868 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9858 HYPMIX: Hyperbolic Interpolative Data Augmentation Ramit Sawhney†‡∗, Megh Thakkar§∗, Shivam Agarwal‡, Di Jin⋆, Diyi Yang△, Lucie Flek‡ †ShareChat ‡Conversational AI and Social Analytics (CAISA) Lab, University of Marburg §BITS, Pilani ⋆Amazon Alexa AI △Georgia Institute of Technology ramitsawhney@sharechat.co, lucie.flek@uni-marburg.de Abstract Interpolation-based regularisation methods for data augmentation have proven to be effec- tive for various tasks and modalities. These methods involve performing mathematical op- erations over the raw input samples or their latent states representations - vectors that of- ten possess complex hierarchical geometries. However, these operations are performed in the Euclidean space, simplifying these repre- sentations, which may lead to distorted and noisy interpolations. We propose HypMix, a novel model-, data-, and modality-agnostic in- terpolative data augmentation technique oper- ating in the hyperbolic space, which captures the complex geometry of input and hidden state hierarchies better than its contemporaries. We evaluate HypMix on benchmark and low resource datasets across speech, text, and vi- sion modalities, showing that HypMix con- sistently outperforms state-of-the-art data aug- mentation techniques. In addition, we demon- strate the use of HypMix in semi-supervised settings. We further probe into the adversar- ial robustness and qualitative inferences we draw from HypMix that elucidate the efficacy of the Riemannian hyperbolic manifolds for interpolation-based data augmentation. 1 Introduction Deep learning methods have improved the state-of- the-art in a wide range of tasks. Yet, when only lim- ited training data is available, they are prone to over- fitting (Zou and Gu, 2019). Numerous data aug- mentation techniques have been proposed, which involve performing operations such as cropping or rotation (Lecun et al., 1998), or paraphrasing (Ku- mar et al., 2019) individual examples. However, ∗Equal contribution. these methods are modality- or dataset-dependent and require domain expertise. Compared to such alteration-based methods, interpolation-based ap- proaches such as Mixup (Zhang et al., 2018) have shown improved performance and generalizability across different modalities. Mixup generates vir- tual training samples from convex combinations of individual inputs and labels to expand the train- ing distribution. Performing Mixup over the latent representations of inputs has led to further improve- ments, as the hidden states of deep neural networks carry more information than raw input samples, (Verma et al., 2019a; Chen et al., 2020a). However, most data augmentation methods can only utilize existing labeled data. Semi-supervised learning methods, on the other hand, can leverage unlabeled data for training. Sev- eral semi-supervised methods use interpolation based regularization methods over unlabeled sam- ples to predict soft labels, and combine them with existing labeled samples to increase the overall training data (Verma et al., 2019b; Chen et al., 2020b). Semi-supervised methods use consistency based regularization training (Miyato et al., 2019) which makes the model predictions robust to per- turbations on unlabeled samples. However, current semi-supervised learning methods do not general- ize across modalities or datasets. Existing data-augmentation and semi-supervised learning methods operate in the Euclidean space, which is a …

Content truncated to 500 words