Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification

Proceedings of the 27th International Conference on Computational Linguistics, pages 2033–2043 Santa Fe, New Mexico, USA, August 20-26, 2018. 2033 Adaptive Learning of Local Semantic and Global Structure Representations for Text Classification Jianyu Zhao1,2,*, Zhiqiang Zhan1,2,*, Qichuan Yang3, Yang Zhang4, Changjian Hu4, Zhensheng Li4, Liuxin Zhang4, and Zhiqiang He4 1University of Chinese Academy of Sciences, Beijing 100049, China 2Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China 3Beihang University, Beijing 100103, China 4Lenovo Research, Beijing 100085, China zhaojy7ict@gmail.com Abstract Representation learning is a key issue for most Natural Language Processing (NLP) tasks. Most existing representation models either learn little structure information or just rely on pre-defined structures, leading to degradation of performance and generalization capability. This paper fo- cuses on learning both local semantic and global structure representations for text classification. In detail, we propose a novel Sandwich Neural Network (SNN) to learn semantic and structure representations automatically without relying on parsers. More importantly, semantic and struc- ture information contribute unequally to the text representation at corpus and instance level. To solve the fusion problem, we propose two strategies: Adaptive Learning Sandwich Neural Net- work (AL-SNN) and Self-Attention Sandwich Neural Network (SA-SNN). The former learns the weights at corpus level, and the latter further combines attention mechanism to assign the weights at instance level. Experimental results demonstrate that our approach achieves competi- tive performance on several text classification tasks, including sentiment analysis, question type classification and subjectivity classification. Specifically, the accuracies are MR (82.1%), SST-5 (50.4%), TREC (96%) and SUBJ (93.9%). 1 Introduction Representation learning plays an important role in various NLP tasks, especially in text classification (Bengio et al., 2013; Le and Mikolov, 2014). Existing representation models for text classification can be categorized into four types: bag-of-words representation models, sequence representation mod- els, structure representation models and attention-based models. Bag-of-words representation models (Salton et al., 1975) are able to represent the sentence accounting for the different words, but fail to encode word order and syntactic structures. Sequence representation models (Kim, 2014; Kalchbrenner et al., 2014) consider word order without using structure information. Structure representation models, such as Tree-LSTM (Tai et al., 2015) and DSCNN (Zhang et al., 2016), take the structure information into account. Attention-based models (Yang et al., 2017; Lin et al., 2017) use attention mechanism to build representations by scoring input words respectively. However, most structure representation methods either learn little structure information or rely heavily on parsers, leading to relatively low performance or gradient vanishing problem. Moreover, to our best knowledge, no one has considered effective fusion of semantic and structure information. These hinder the performance of sentence representation for text classification. To address these issues, we propose a novel model named Sandwich Neural Network, which con- sists of a Convolutional Neural Network (CNN) layer in the middle of two Long Short-Term Memory (LSTM) layers like a sandwich. We define local semantic representation as n-grams feature and global structure representation as the dependency relationship between words or phrases in the sentence. SNN can generate …

Content truncated to 500 words