Augmenting NLP models using Latent Feature Interpolations

Proceedings of the 28th International Conference on Computational Linguistics, pages 6931–6936 Barcelona, Spain (Online), December 8-13, 2020 6931 Augmenting NLP models using Latent Feature Interpolations Amit Jindal* Manipal Academy of Higher Education amitj646@gmail.com Arijit Ghosh Chowdhury* Manipal Academy of Higher Education arijit10@gmail.com Aniket Didolkar* Manipal Academy of Higher Education adidolkar123@gmail.com Di Jin MIT jindi@mit.edu Ramit Sawhney NSIT, New Delhi ramits.co@nsit.net.in Rajiv Ratn Shah MIDAS, IIIT Delhi rajivratn@iiitd.ac.in Abstract Models with a large number of parameters are prone to over-fitting and often fail to capture the underlying input distribution. We introduce Emix, a data augmentation method that uses interpo- lations of word embeddings and hidden layer representations to construct virtual examples. We show that Emix shows significant improvements over previously used interpolation based regular- izers and data augmentation techniques. We also demonstrate how our proposed method is more robust to sparsification. We highlight the merits of our proposed methodology by performing thorough quantitative and qualitative assessments. 1 Introduction In this paper, we deal with the problem of sentence classification. Current state-of-the-art deep learning approaches for sentence classification have millions of parameters, thus requiring a large number of training examples, which may be time-consuming and expensive to obtain. We present a new data augmentation technique called Emix to improve the performance of current text classification models. In general, it is difficult to come up with rules for language transformation similar to image transfor- mations; hence universal data augmentation techniques have not thoroughly been explored in NLP yet (Wei and Zou, 2019). One such technique is Mixup - it uses systematic transformations to make sure the model trains on samples from the vicinity distribution (Chapelle et al., 2001) along with the original distribution of the training data. Mixup is a data-agnostic data augmentation method that is proven to be effective in Computer Vision tasks (Zhang et al., 2017; Verma et al., 2018). Guo et al. (2019) extends Mixup to Text classification using interpolations in the embedding space. In our proposed method, we eliminate the common mean vector from the embeddings. We then interpolate hidden representations at multiple layers, including the embedding layer. This mixing method is also simple and easy to implement and leads to a further improvement in performance. Related Work: Data Augmentation has been vastly used in Computer Vision (Krizhevsky et al., 2012), which employs techniques like flipping, scaling, and rotation. Data augmentation has also been explored in NLP to some extent. (Zhang et al., 2015) replaced words with their synonyms according to a geometric distribution. (Wang and Yang, 2015) used k-NN and Cosine Similarity metrics to find a similar word for replacement. (Wei and Zou, 2019) introduced EDA (Easy Data Augmentation) for NLP which includes Synonym Replacement, Random Insertion, Random Deletion, and Random Swap. These methods have the risk of completely changing the context of the sentence with the replacement of a word which alters the entire This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 6932 meaning of the sentence. We create …

Content truncated to 500 words