Investigating the Working of Text Classifiers

Proceedings of the 27th International Conference on Computational Linguistics, pages 2120–2131 Santa Fe, New Mexico, USA, August 20-26, 2018. 2120 Investigating the Working of Text Classifiers Devendra Singh Sachan♠,△, Manzil Zaheer♠, Ruslan Salakhutdinov♠ ♠School of Computer Science, Carnegie Mellon University △Petuum, Inc Pittsburgh, PA, USA {dsachan, manzilz, rsalakhu}@andrew.cmu.edu Abstract Text classification is one of the most widely studied tasks in natural language processing. Mo- tivated by the principle of compositionality, large multilayer neural network models have been employed for this task in an attempt to effectively utilize the constituent expressions. Almost all of the reported work train large networks using discriminative approaches, which come with a caveat of no proper capacity control, as they tend to latch on to any signal that may not generalize. Using various recent state-of-the-art approaches for text classification, we explore whether these models actually learn to compose the meaning of the sentences or still just focus on some keywords or lex- icons for classifying the document. To test our hypothesis, we carefully construct datasets where the training and test splits have no direct overlap of such lexicons, but overall language structure would be similar. We study various text classifiers and observe that there is a big performance drop on these datasets. Finally, we show that even simple models with our proposed regularization techniques, which disincentivize focusing on key lexicons, can substantially improve classification accuracy. 1 Introduction Text classification is one of the fundamental tasks in natural language processing (NLP) in which the objective is to categorize text documents into one of the predefined classes. This task has a lot of applications such as topic classification of news articles, sentiment analysis of reviews, email filtering, etc. Text classification still remains a significant challenge in language understanding as we need to encode the intrinsic grammatical relations between sentences in the semantic meaning of a document. This is especially crucial for sentiment classification because relations like “contrast” and “cause” can have great influence on determining the meaning and the overall polarity of a document. Traditionally, for text classification, bag-of-words model (Harris, 1954) was used to represent a document. This simple approach uses the term frequency as features followed by a classifier such as Naïve Bayes (McCallum and Nigam, 1998) or Support Vector Machine (SVM) (Joachims, 1998). One drawback of this approach is that it ignores the word order and grammatical structure. It also suffers from data sparsity problem when the training set’s size is small but has shown to give good results when size is not an issue (Wang and Manning, 2012). Before applying these methods, feature selection is typically carried out to reduce the effective vocabulary size by removing the noisy features (Manning et al., 2008). A key property of these linear classifiers is that they assign high weights to some class label specific keywords, which are also known as lexicons. The next generation of approaches includes neural networks that have shown to outperform bag-of- words models in text classification tasks (Kim, 2014; Johnson and Zhang, 2016). …

Content truncated to 500 words