Weighted Neural Bag-of-n-grams Model: New Baselines for Text Classification

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1591–1600, Osaka, Japan, December 11-17 2016. Weighted Neural Bag-of-n-grams Model: New Baselines for Text Classification Bofang Li∗ , Zhe Zhao∗, Tao Liu, Puwei Wang†, Xiaoyong Du School of Information, Renmin University of China Key laboratory of Data Engineering and Knowledge Engineering, MOE {libofang,helloworld,tliu,wangpuwei,duyong}@ruc.edu.cn Abstract NBSVM is one of the most popular methods for text classification and has been widely used as baselines for various text representation approaches. It uses Naive Bayes (NB) feature to weight sparse bag-of-n-grams representation. N-gram captures word order in short context and NB fea- ture assigns more weights to those important words. However, NBSVM suffers from sparsity problem and is reported to be exceeded by newly proposed distributed (dense) text representa- tions learned by neural networks. In this paper, we transfer the n-grams and NB weighting to neural models. We train n-gram embeddings and use NB weighting to guide the neural models to focus on important words. In fact, our methods can be viewed as distributed (dense) counter- parts of sparse bag-of-n-grams in NBSVM. We discover that n-grams and NB weighting are also effective in distributed representations. As a result, our models achieve new strong baselines on 9 text classification datasets, e.g. on IMDB dataset, we reach performance of 93.5% accuracy, which exceeds previous state-of-the-art results obtained by deep neural models. All source codes are publicly available at https://github.com/zhezhaoa/neural_BOW_toolkit. 1 Introduction Text representation is a core technology for many NLP tasks. Most text representation approaches fall into one of the two classes: sparse and distributed (dense) representations. One of the most popular sparse representations is bag-of-words (BOW), where each dimension represents the number of occurrences of a word in a text. Though simple, BOW enjoys the advantages of being efficient and surprisingly effective. Until now, BOW representation still serves as baselines on a range of NLP tasks. In distributed representation, texts are represented by low-dimensional real vectors. Recently, there has been a surge of work proposing to learn distributed text representation through neural networks. There exists two lines of researches in neural models. The first is order/syntax-aware models, such as Convo- lutional Neural Networks (CNNs) (Kim, 2014; Kalchbrenner et al., 2014; Zhang et al., 2016), Recurrent Neural Networks (RNNs) (Dai and Le, 2015) and Recursive Neural Networks (RecNNs) (Socher et al., 2011; Socher et al., 2012). In these models, words are firstly embedded into low-dimensional real vec- tors (word embeddings) as the input, and then order/syntax-aware compositions upon words are learned by neural networks (Goldberg, 2015). Another line is neural bag-of-words models, where unordered compositions are learned upon word embeddings (Iyyer et al., 2015). The simplest neural bag-of-words model is word embeddings average. Compared to order/syntax-aware models, they are much more ef- ficient in training (Iyyer et al., 2015) but lose the accuracies due to the ignorance of the order/syntax information. To the best of our knowledge, most of the neural models map isolated words (uni-grams) to embed- dings. Sometimes, consecutive words …

Content truncated to 500 words