Enhancing Local Feature Extraction with Global Representation for Neural Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 496–506, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 496 Enhancing Local Feature Extraction with Global Representation for Neural Text Classification Guocheng Niu1*, Hengru Xu2*†, Bolei He1, Xinyan Xiao1, Hua Wu1, Sheng Gao2 1 Baidu Inc., Beijing, China 2 Beijing University of Posts and Telecommunications 1{niuguocheng, hebolei, xiaoxinyan, wu hua}@baidu.com 2{xuhengru, gaosheng}@bupt.edu.cn Abstract For text classification, traditional local fea- ture driven models learn long dependency by deeply stacking or hybrid modeling. This pa- per proposes a novel Encoder1-Encoder2 ar- chitecture, where global information is incor- porated into the procedure of local feature extraction from scratch. In particular, En- coder1 serves as a global information provider, while Encoder2 performs as a local feature extractor and is directly fed into the classi- fier. Meanwhile, two modes are also designed for their interactions. Thanks to the aware- ness of global information, our method is able to learn better instance specific local features and thus avoids complicated upper operations. Experiments conducted on eight benchmark datasets demonstrate that our proposed archi- tecture promotes local feature driven models by a substantial margin and outperforms the previous best models in the fully-supervised setting. 1 Introduction Text classification is a fundamental task in natural language processing, which is widely used in vari- ous applications such as spam detection, sentiment analysis and topic classification. One of the main- stream approaches firstly utilizes explicit local ex- tractors to identity key local patterns and classifies based on them afterwards. In this paper, we call this line of research as local feature driven mod- els. Lots of proposed methods can be grouped into this scope. Ngrams have been tradition- ally exploited in statistical machine learning ap- proaches (Pang et al., 2002; Wang and Manning, 2012). For deep neural networks, encoding local features into low-dimensional distributed ngrams ?These authors contributed equally to this work. †This work was done while the author was an intern at Baidu Inc. Case1: Apple is really amazing! I am fed up to carry my clunky camera. Case2: Apple is famous around world and de- serves to be called “nutritional powerhouses”. Table 1: Topic classification examples for Technology and Health, where Apple is ambiguous within local context. embeddings (Joulin et al., 2016; Qiao et al., 2018) and simply bagging of them have been proved ef- fective and highly efficient. Convolutional Neu- ral Networks (CNN) (LeCun et al., 2010) are promising methods for their strong capacities in capturing local invariant regularities (Kim, 2014). More recently, Wang (2018) proposes the Discon- nected Recurrent Neural Network (DRNN), which utilizes RNN to extract local features for larger windows and has reported best results on several benchmarks. Despite having good interpretability and re- markable performance, current local feature ex- traction still has one shortcoming. As shown in Table 1, the real meaning of Apple can only be correctly recognized from overall view instead of narrow window. If the …

Content truncated to 500 words