Multi-hop Graph Convolutional Network with High-order Chebyshev Approximation for Text Reasoning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6563–6573 August 1–6, 2021. ©2021 Association for Computational Linguistics 6563 Multi-hop Graph Convolutional Network with High-order Chebyshev Approximation for Text Reasoning Shuoran Jiang, Qingcai Chen∗, Xin Liu, Baotian Hu, Lisai Zhang Harbin Institute of Technology, Shenzhen {shuoran.chiang, hit.xinliu, lisaizhang2016}@gmail.com {qingcai.chen, hubaotian}@hit.edu.cn Abstract Graph convolutional network (GCN) has be- come popular in various natural language pro- cessing (NLP) tasks with its superiority in long- term and non-consecutive word interactions. However, existing single-hop graph reason- ing in GCN may miss some important non- consecutive dependencies. In this study, we define the spectral graph convolutional network with the high-order dynamic Chebyshev ap- proximation (HDGCN), which augments the multi-hop graph reasoning by fusing messages aggregated from direct and long-term depen- dencies into one convolutional layer. To alle- viate the over-smoothing in high-order Cheby- shev approximation, a multi-vote based cross- attention (MVCAttn) with linear computation complexity is also proposed. The empiri- cal results on four transductive and inductive NLP tasks and the ablation study verify the efficacy of the proposed model. Our source code is available at https://github.com/ MathIsAll/HDGCN-pytorch. 1 Introduction Graph neural networks (GNNs) are usually used to learn the node representations in Euclidean space from graph data, which have been developed to one of the hottest research topics in recent years (Zhang, 2020). The primitive GNNs relied on recur- sive propagation on graphs, which takes a long time to train (Zhang et al., 2019b). One major variant of GNNs, graph convolutional networks (GCNs) (Kipf and Welling, 2017; Yao et al., 2019), takes spectral filtering to replace recursive message pass- ing and needs only a shallow network to convergent, which have been used in various NLP tasks. For example, Yao et al. (2019) constructed the text as a graph and input it to a GCN. This method achieved better results than conventional deep learning mod- els in text classification. Afterward, the GCNs ∗corresponding author: Qingcai Chen have became popular in more tasks, such as word embedding (Zhang et al., 2020b), semantic anal- ysis (Zhang et al., 2019a), document summariza- tion (Wang et al., 2020), knowledge graph (Wang et al., 2018), etc. The spectral graph convolution in Yao’s GCN is a localized first-order Chebyshev approximation. It is equal to a stack of 1-step Markov chain (MC) layer and fully connected (FC) layer. Unlike the multi-step Markov chains, the message propaga- tion in vanilla GCN lacks the node probability tran- sitions. As a result, the multi-hop graph reason- ing is very tardy in GCN and easily causes the suspended animation problem (Zhang and Meng, 2019). However, the probability transition on the graph is useful to improve the efficiency in learning contextual dependencies. In many NLP tasks (like the question answering (QA) system and entity re- lation extraction), the features of the two nodes need to be aligned. As an example, Figure 1 shows a simple graph where the node n4 is a pronoun …

Content truncated to 500 words