Cascaded Semantic and Positional Self-Attention Network for Document Classification

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 669–677 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 669 Abstract Transformers have shown great success in learning representations for language modelling. However, an open challenge still remains on how to systematically aggregate semantic information (word embedding) with positional (or temporal) information (word orders). In this work, we propose a new architecture to aggregate the two sources of information using cascaded semantic and positional self-attention network (CSPAN) in the context of document classification. The CSPAN uses a semantic self-attention layer cascaded with Bi-LSTM to process the semantic and positional information in a sequential manner, and then adaptively combine them together through a residual connection. Compared with commonly used positional encoding schemes, CSPAN can exploit the interaction between semantics and word positions in a more interpretable and adaptive manner, and the classification performance can be notably improved while simultaneously preserving a compact model size and high convergence rate. We evaluate the CSPAN model on several benchmark data sets for document classification with careful ablation studies, and demonstrate the encouraging results compared with state of the art. 1 Introduction Document classification is one of the fundamental problems in natural language processing, which is aimed at assigning one or multiple labels to a (typically) short text paragraph. Wide applications can be found in sentiment analysis (Moraes et al., 2013; Tang et al., 2015)，subject categorization * Corresponding author. (Wang et al., 2012),spam email detection (Sahami et al., 1998) and doc1ument ranking (Wang et al., 2014). In recent years, deep neural networks have shown great potential in document classification and updated state-of-the-art performance. Popular approaches include Recurrent neural networks (RNN) (Yogatama et al., 2017), convolutional neural networks (CNN) (Zhang et al., 2015) and Attention-based methods (Transformers) (Gong et al., 2019; Adhikari et al., 2019), or a mixture of them. Different lines of methods have their respective pros and cons. For example, RNNs are highly effective models for exploiting word orders in learning useful representations, thanks to the iterative update of the hidden states that depend on both the semantics of the current word and that of historical words (or a concise summary of them), and the long-range dependency made possible through LSTMs (Yang et al., 2016; Stephen et al., 2018; Adhikari et al., 2019). Of course, the sequential processing nature makes it less efficient computationally. CNNs have gained huge success in image procesing and classification and were recently introduced to NLP domains like document classification (Zhang et al., 2015; Lei et al.,2015; Conneau et al., 2016; Kim and Yang, 2018; Kim, 2014).The local convolutional operator is sensitive to word orders but only partially and limited by the size of the kernel, and so long-term relations may need many layers and therefore be challenging. Transformers, different from both, fully exploit the modelling power of self-attention mechanism (Shen et al., 2018; Gao et al., 2018; Zheng et al., 2018) and have significantly improved state of the art in many NLP tasks such as machine translation …

Content truncated to 500 words