From Random to Supervised: A Novel Dropout Mechanism Integrated with Global Information

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 573–582 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 573 From Random to Supervised: A Novel Dropout Mechanism Integrated with Global Information Hengru Xu1, † Shen Li2, † Renfen Hu3 Si Li1 Sheng Gao1, ‡ 1{xuhengru, lisi, gaosheng}@bupt.edu.cn 2shen@deeplycurious.ai 3irishere@mail.bnu.edu.cn 1 SICE, Beijing University of Post and Telecommunication 2 Deeplycurious.ai 3 Institute of Chinese Information Processing, Beijing Normal University Abstract Dropout is used to avoid overfitting by ran- domly dropping units from the neural net- works during training. Inspired by dropout, this paper presents GI-Dropout, a novel dropout method integrating with global infor- mation to improve neural networks for text classification. Unlike the traditional dropout method in which the units are dropped ran- domly according to the same probability, we aim to use explicit instructions based on global information of the dataset to guide the training process. With GI-Dropout, the model is sup- posed to pay more attention to inapparent fea- tures or patterns. Experiments demonstrate the effectiveness of the dropout with global infor- mation on seven text classification tasks, in- cluding sentiment analysis and topic classifi- cation. 1 Introduction Recently, neural networks have achieved remark- able results in natural language processing (NLP). Convolutional Neural Network (CNN) and Re- current Neural Network (RNN) are two popular types of neural network architectures and both of them are widely applied to various NLP tasks. CNN is known for its strong ability in extracting position-invariant features and RNN is highlighted in modeling sequences (Yin et al., 2017). In sen- tence classification tasks, models based on CNN or RNN aim to represent sentences as appropri- ate embeddings, which are supposed to encode se- mantic features for the classification. However, with the consideration of computa- tional complexity and spatial limitation, neural networks are often trained via mini-batch in which global information is gathered implicitly rather work.† Hengru Xu and Shen Li contributed equally to this ‡ Corresponding author. than explicitly. To facilitate the learning process, Li et al. (2017) extract global semantic features from the training dataset, and encode them into CNN filters with a novel initialization mechanism. This approach gains significant improvements in sentiment analysis and topic classification tasks. Unlike most of machine learning methods, the advantage of neural networks is extracting features with less need of feature engineering. In gen- eral, the stronger ability of a model to learn fea- tures automatically, the better performance it will achieve. However, during the training process, neural networks tend to focus on some distinctive words or phrases but ignore other noteworthy pat- terns, which may result in overfitting, especially in a small dataset. To avoid this problem, dropout is proposed (Hinton et al., 2012; Srivastava et al., 2014). The key idea of dropout is to randomly drop units from the neural network during train- ing and use a smaller weight of these units in the test. Inspired by the above works, we propose a novel dropout method guided …

Content truncated to 500 words