MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification

Proceedings of NAACL-HLT 2016, pages 1522–1527, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification Ye Zhang1 Stephen Roller1 Byron C. Wallace2 1Department of Computer Science 2iSchool The University of Texas at Austin The University of Texas at Austin {yezhang,roller}@cs.utexas.edu byron.wallace@utexas.edu Abstract We introduce a novel, simple convolution neu- ral network (CNN) architecture – multi-group norm constraint CNN (MGNC-CNN) – that capitalizes on multiple sets of word embed- dings for sentence classification. MGNC- CNN extracts features from input embedding sets independently and then joins these at the penultimate layer in the network to form a fi- nal feature vector. We then adopt a group regularization strategy that differentially pe- nalizes weights associated with the subcom- ponents generated from the respective embed- ding sets. This model is much simpler than comparable alternative architectures and re- quires substantially less training time. Fur- thermore, it is flexible in that it does not re- quire input word embeddings to be of the same dimensionality. We show that MGNC-CNN consistently outperforms baseline models. 1 Introduction Neural models have recently gained popularity for Natural Language Processing (NLP) tasks (Gold- berg, 2015; Collobert and Weston, 2008; Cho, 2015). For sentence classification, in particular, Convolution Neural Networks (CNN) have realized impressive performance (Kim, 2014; Zhang and Wallace, 2015). These models operate over word embeddings, i.e., dense, low dimensional vector rep- resentations of words that aim to capture salient se- mantic and syntactic properties (Collobert and We- ston, 2008). An important consideration for such models is the specification of the word embeddings. Several op- tions exist. For example, Kalchbrenner et al. (2014) initialize word vectors to random low-dimensional vectors to be fit during training, while Johnson and Zhang (2014) use fixed, one-hot encodings for each word. By contrast, Kim (2014) initializes word vectors to those estimated via the word2vec model trained on 100 billion words of Google News (Mikolov et al., 2013); these are then up- dated during training. Initializing embeddings to pre-trained word vectors is intuitively appealing be- cause it allows transfer of learned distributional se- mantics. This has allowed a relatively simple CNN architecture to achieve remarkably strong results. Many pre-trained word embeddings are now read- ily available on the web, induced using different models, corpora, and processing steps. Different embeddings may encode different aspects of lan- guage (Pad´o and Lapata, 2007; Erk and Pad´o, 2008; Levy and Goldberg, 2014): those based on bag-of- words (BoW) statistics tend to capture associations (doctor and hospital), while embeddings based on dependency-parses encode similarity in terms of use (doctor and surgeon). It is natural to consider how these embeddings might be combined to improve NLP models in general and CNNs in particular. Contributions. We propose MGNC-CNN, a novel, simple, scalable CNN architecture that can accom- modate multiple off-the-shelf embeddings of vari- able sizes. Our model treats different word em- beddings as distinct groups, and applies CNNs in- dependently to each, thus generating corresponding feature vectors (one …

Content truncated to 500 words