Text Graph Transformer for Document Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8322–8327, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8322 Text Graph Transformer for Document Classification Haopeng Zhang Jiawei Zhang IFM Lab, Department of Computer Science, Florida State University, FL, USA haopeng,jiawei@ifmlab.org Abstract Text classification is a fundamental problem in natural language processing. Recent stud- ies applied graph neural network (GNN) tech- niques to capture global word co-occurrence in a corpus. However, previous works are not scalable to large-sized corpus and ignore the heterogeneity of the text graph. To ad- dress these problems, we introduce a novel Transformer based heterogeneous graph neu- ral network, namely Text Graph Transformer (TG-Transformer). Our model learns effec- tive node representations by capturing struc- ture and heterogeneity from the text graph. We propose a mini-batch text graph sampling method that significantly reduces computing and memory costs to handle large-sized corpus. Extensive experiments have been conducted on several benchmark datasets, and the results demonstrate that TG-Transformer outperforms state-of-the-art approaches on text classifica- tion task. 1 Introduction Text classification is a widely studied problem in natural language processing and has been ad- dressed in many real-world applications such as news filtering, spam detection, and health record systems (Kowsari et al., 2019; Che et al., 2015; Zhang et al., 2018). The objective is to assign corresponding labels to textual units based on text representations. Deep learning models like Convolutional Neural Networks (CNN) (Kim, 2014) and Recurrent Neu- ral Networks (RNN) (Hochreiter and Schmidhu- ber, 1997) have been applied for text representation learning instead of traditional hand-crafted features, such as n-gram and bag-of-words (BoW) (Joulin et al., 2016). Researchers have recently turned to Graph Neural Network (GNN) to exploit global fea- tures in text representation learning, which learns node embedding by aggregating information from neighbors through edges. Defferrard et al. (2016) first generalized CNN to graph for text classifica- tion task. Then Yao et al. (2019) applied Graph Convolution Network (GCN) (Kipf and Welling, 2016) on a corpus level heterogeneous text graph and achieved state-of-the-art performance. Liu et al. (2020) further improved classification accuracy by expanding the text graph with semantic and syntac- tic contextual information. However, these GCN-based models on heteroge- neous text graphs suffer from two practical issues. Firstly, none of these models are scalable to large- sized corpus due to high computation and memory costs. Calculation of all the nodes in the graph is required at each layer during training. Secondly, all these models ignore the heterogeneity of the text graph, which consists of both document and word nodes. Distinguishing nodes of different types will benefit node representation learning. To address the above problems, we pro- pose a novel Transformer-based heterogeneous GNN model, namely Text Graph Transformer (TG-Transformer). Instead of learning based on the full text graph, we propose a text graph sampling method that enables subgraph mini-batch training. The significantly reduced computing and memory costs make the model scalable to large-sized cor- pus. Moreover, we utilize Transformer to aggregate information in subgraph …

Content truncated to 500 words