Initializing Convolutional Filters with Semantic Features for Text Classification

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1884–1889 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Initializing Convolutional Filters with Semantic Features for Text Classification Shen Li1,2 shen@mail.bnu.edu.cn Zhe Zhao4,5 helloworld@ruc.edu.cn Tao Liu4,5 tliu@ruc.edu.cn Renfen Hu3,† irishere@mail.bnu.edu.cn Xiaoyong Du4,5 duyong@ruc.edu.cn 1 Institute of Chinese Information Processing, Beijing Normal University 2 UltraPower-BNU Joint Laboratory for Artificial Intelligence, Beijing Normal University 3 College of Chinese Language and Culture, Beijing Normal University 4 School of Information, Renmin University of China 5 Key Laboratory of Data Engineering and Knowledge Engineering, MOE Abstract Convolutional Neural Networks (CNNs) are widely used in NLP tasks. This pa- per presents a novel weight initialization method to improve the CNNs for text clas- sification. Instead of randomly initializ- ing the convolutional filters, we encode se- mantic features into them, which helps the model focus on learning useful features at the beginning of the training. Experiments demonstrate the effectiveness of the ini- tialization technique on seven text classifi- cation tasks, including sentiment analysis and topic classification. 1 Introduction Recently, neural networks (NNs) dominate the state-of-the-art results on a wide range of natu- ral language processing (NLP) tasks. The com- monly used neural networks in NLP include Re- current NNs, Convolutional NNs, Recursive NNs and their combinations. NNs are known for their strong abilities to learn features automatically. However, the lack of data or inappropriate param- eter settings might greatly limit the generalization abilities of the models (Bengio et al., 2009; Le- Cun et al., 2015; Krizhevsky et al., 2012; Srivas- tava et al., 2014). To enhance the performance, a lot of improved methods have been proposed, e.g. developing advanced structures (Zhao et al., 2015; Zhang et al., 2016a), introducing prior knowledge (Hu et al., 2016) and utilizing external resources (Xie et al., 2016; Qian et al., 2016). It is also noteworthy that the neural networks’ performance is sensitive to weight initialization † Corresponding author. because their objectives are non-convex (Glorot and Bengio, 2010; Saxe et al., 2013; Mishkin and Matas, 2015). In fact, initialization techniques even play a role of catalyst for the revival of neu- ral networks (Hinton et al., 2006; LeCun et al., 2015). Most improvements on initializing weights are based on mathematical methods, e.g. xavier initialization (Glorot and Bengio, 2010) and or- thogonal initialization (Saxe et al., 2013). For NLP tasks, an influential technique is to use pre- trained word vectors to initialize embedding layers (Kim, 2014; Chen and Manning, 2014). Consider the embedding layers could be initialized by pre- trained word vectors, how about weights in other layers that are still randomly initialized? Inspired by this question, we propose a sim- ple yet effective method to improve CNNs by ini- tializing convolutional layers (filters). Unlike the previous weight initialization based on mathemati- cal methods, we encode semantic features into the filters instead of initializing them randomly. As CNNs exploit 1-D convolutional filters to extract n-gram features, our method aims at helping the filters focus on learning useful n-grams, …

Content truncated to 500 words