ACENet: Attention Guided Commonsense Reasoning on Hybrid Knowledge Graph

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 890 - 899 December 7-11, 2022 ©2022 Association for Computational Linguistics DropMix: A Textual Data Augmentation Combining Dropout with Mixup Fanshuang Kong1, Richong Zhang1, 2 ∗, Xiaohui Guo3, Samuel Mensah4, Yongyi Mao5 1SKLSDE, Beihang University, Beijing, China 2Zhongguancun Laboratory, Beijing, China 3Hangzhou Innovation Institute, Beihang University, Hangzhou, China 4Department of Computer Science, University of Sheffield, UK 5School of Electrical Engineering and Computer Science, University of Ottawa, Canada {kongfs,zhangrc,guoxh}@act.buaa.edu.cn s.mensah@sheffield.ac.uk, ymao@uottawa.ca Abstract Overfitting is a common problem when there is insufficient data to train deep neural networks in machine learning tasks. Data augmenta- tion regularization methods such as Dropout, Mixup, and their enhanced variants, are effec- tive and prevalent, and achieve promising per- formance to overcome overfitting. However, in text learning, most of the existing regular- ization approaches merely adopt ideas from computer vision without considering the im- portance of dimensionality in natural language processing. In this paper, we argue that the property is essential to overcome overfitting in text learning. Accordingly, we present a saliency map informed textual data augmenta- tion and regularization framework, which com- bines Dropout and Mixup, namely DropMix, to mitigate the overfitting problem in text learning. In addition, we design a procedure that drops and patches fine grained shapes of the saliency map under the DropMix framework to enhance regularization. Empirical studies confirm the effectiveness of the proposed approach on 12 text classification tasks. 1 Introduction Deep neural networks have shown its effective- ness, achieving state-of-the-art result in many natu- ral language processing (NLP) tasks. However, these models are likely to overfit training sets with few samples due to the large number of pa- rameters they contain. Regularization techniques, such as weighted decay (Krogh and Hertz, 1992), dropout (Srivastava et al., 2014), and data augmen- tation (Goodfellow et al., 2015; Zhang et al., 2017) are powerful to overcome overfitting by making slight modifications on the neural network or data space to improve generlization. Data augmentation methods have shown to be useful for overcoming overfitting by increasing the ∗Corresponding author XA Food in this restaurant is delicious P XB The environment is bad N XC Food in this restaurant bad delicious 0.83P Table 1: Illustration of an example XC generated by SSMix. The darker the red (or blue) shade is, the more salient the words in the positive (or negative) sentiment decision. SSMix uses a Saliency Map (Simonyan et al., 2013) to measure the saliency of words. SSMix sub- stitute the least salient word “is” in XA with the most salient “bad” in XB. The mixup ratio λ = 0.17 as the length of the salient words (i.e., “bad”) is one out of six words in XC. Thus, the generated label is 17% negative or 83% positive. training dataset size via adding augmented ver- sions of already existing training data. For ex- ample, Goodfellow et al. (2015) adds the worst case adversarial sample into the training dataset; Mixup (Zhang et al., 2017) introduces new …

Content truncated to 500 words