BertGCN: Transductive Text Classification by Combining GNN and BERT

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1456–1462 August 1–6, 2021. ©2021 Association for Computational Linguistics 1456 BertGCN: Transductive Text Classification by Combining GCN and BERT Yuxiao Lin♠, Yuxian Meng♣, Xiaofei Sun♣ Qinghong Han♣, Kun Kuang♠, Jiwei Li♠♣ and Fei Wu♠ ♠Computer Science Department, Zhejiang University ♣ ShannonAI {yuxiaolinling, kunkuang, jiwei_li, wufei}@zju.edu.cn {yuxian_meng, xiaofei_sun, qinghong_han}@shannonai.com Abstract In this work, we propose BertGCN, a model that combines large scale pretraining and trans- ductive learning for text classification. Bert- GCN constructs a heterogeneous graph over the dataset and represents documents as nodes using BERT representations. By jointly train- ing the BERT and GCN modules within Bert- GCN, the proposed model is able to lever- age the advantages of both worlds: large-scale pretraining which takes the advantage of the massive amount of raw data and transductive learning which jointly learns representations for both training data and unlabeled test data by propagating label influence through graph convolution. Experiments show that BertGCN achieves SOTA performances on a wide range of text classification datasets.1 1 Introduction Text classification is a core task in natural language processing (NLP) and has been used in many real- world applications such as spam detection (Wang, 2010) and opinion mining (Bakshi et al., 2016). Transductive learning (Vapnik, 1998) is a particular method for text classification which makes use of both labeled and unlabeled examples in the train- ing process. Graph neural networks (GNNs) serve as an effective approach for transductive learning (Yao et al., 2019; Liu et al., 2020). In these works, a graph is constructed to model the relationship be- tween documents. Nodes in the graph represent text units such as words and documents, while edges are constructed based on the semantic similarity be- tween nodes. GNNs are then applied to the graph to perform node classification. The merits of GNNs and transductive learning are as follows: (1) the de- cision for an instance (both training and test) does not depend merely on itself, but also its neighbors. 1Code available at https://github.com/ ZeroRin/BertGCN. This makes the model more immune to data out- liers; (2) at the training time, since the model prop- agates influence from supervised labels across both training and test instances through graph edges, unlabeled data also contributes to the process of representation learning, and consequently higher performances. Large-scale pretraining has recently demonstrated their effectiveness on a variety of NLP tasks (De- vlin et al., 2018; Liu et al., 2019). Trained on large-scale unlabeled corpora in an unsupervised manner, large-scale pretrained models are able to learn implicit but rich text semantics in language at scale. Intuitively, large-scale pretrained mod- els have potentials to benefit transductive learning. However, existing models for transductive text clas- sification (Yao et al., 2019; Liu et al., 2020) did not take large-scale pretraining into consideration, and its effectiveness still remains unclear. In this work, we propose BertGCN, a model that combines the advantages of both large-scale pre- training and transductive learning for text clas- sification. BertGCN constructs a heterogeneous graph for …

Content truncated to 500 words