Feature Projection for Improved Text Classification

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8161–8171 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8161 Feature Projection for Improved Text Classification Qi Qin1,2,∗, Wenpeng Hu3,∗, Bing Liu2,† 1 Center for Data Science, AAIS, Peking University 2 Wangxuan Institute of Computer Technology, Peking University 3 Department of Information Science, Peking University {qinqi, wenpeng.hu, dcsliub}@pku.edu.cn Abstract In classification, there are usually some good features that are indicative of class labels. For example, in sentiment classification, words like good and nice are indicative of the posi- tive sentiment and words like bad and terrible are indicative of the negative sentiment. How- ever, there are also many common features (e.g., words) that are not indicative of any spe- cific class (e.g., voice and screen, which are common to both sentiment classes and are not discriminative for classification). Although deep learning has made significant progresses in generating discriminative features through its powerful representation learning, we be- lieve there is still room for improvement. In this paper, we propose a novel angle to fur- ther improve this representation learning, i.e., feature projection. This method projects exist- ing features into the orthogonal space of the common features. The resulting projection is thus perpendicular to the common features and more discriminative for classification. We ap- ply this new method to improve CNN, RNN, Transformer, and Bert based text classification and obtain markedly better results. 1 Introduction Text classification is an important task in natural language processing and text mining. It has a very wide range of applications, such as sentiment classi- fication (Liu, 2012), question classification (Li and Roth, 2002), and deception detection (Liu, 2012; Feng et al., 2012). In recent years, deep learn- ing models have been shown to outperform tra- ditional classification methods (Kim, 2014; Iyyer et al., 2015; Tang et al., 2015; Dai and Le, 2015; Jin et al., 2016; Joulin et al., 2017; Shen et al., 2018). Given the input document, the system applies a mapping function (e.g., averaging or summation, a ∗Equal Contribution. †Corresponding Author. convolution neural network (CNN), recurrent neu- ral network (RNN), and so on) to learn a dense representation of the document and then uses this representation to perform the final classification. Representation learning is one of the key strengthes of deep learning. In this paper, we propose to further improve the representation learning, i.e., to make the represen- tation more discriminative for classification. Note that throughout the paper we will use sentence sen- timent classification as an example to explain dif- ferent ideas, but in our experiments, non-sentiment classification datasets are also used to show the generality of the proposed method. For text clas- sification, many neural networks and embedding techniques have been devised and applied, e.g., RNN, CNN, Transformer (Vaswani et al., 2017) and Bert (Devlin et al., 2018). For example, RNN can model the whole sentence and also capture the long-term dependencies within the sentence. How- ever, modeling the entire sequence may neglect some key local contexts …

Content truncated to 500 words