Heterogeneous Graph Transformer for Graph-to-Sequence Learning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7145–7154 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7145 Heterogeneous Graph Transformer for Graph-to-Sequence Learning Shaowei Yao, Tianming Wang, Xiaojun Wan Wangxuan Institute of Computer Technology, Peking University Center for Data Science, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University {yaosw,wangtm,wanxiaojun}@pku.edu.cn Abstract The graph-to-sequence (Graph2Seq) learning aims to transduce graph-structured represen- tations to word sequences for text generation. Recent studies propose various models to en- code graph structure. However, most previous works ignore the indirect relations between dis- tance nodes, or treat indirect relations and di- rect relations in the same way. In this paper, we propose the Heterogeneous Graph Trans- former to independently model the different re- lations in the individual subgraphs of the origi- nal graph, including direct relations, indirect relations and multiple possible relations be- tween nodes. Experimental results show that our model strongly outperforms the state of the art on all four standard benchmarks of AMR- to-text generation and syntax-based neural ma- chine translation. 1 Introduction Graph-to-sequence (Graph2Seq) learning has at- tracted lots of attention in recent years. Many Nat- ural Language Process (NLP) problems involve learning from not only sequential data but also more complex structured data, such as semantic graphs. For example, AMR-to-text generation is a task of generating text from Abstract Meaning Representation (AMR) graphs, where nodes denote semantic concepts and edges refer to relations be- tween concepts (see Figure 1 (a)). In addition, it has been shown that even if the sequential input can be augmented by additional structural information, bringing benefits for some tasks, such as semantic parsing (Pust et al., 2015; Guo and Lu, 2018) and machine translation (Bastings et al., 2017). There- fore, Xu et al. (2018b) introduced the Graph2Seq problems which aim to generate target sequence from graph-structured data. The main challenge for Graph2Seq learning is to build a powerful encoder which is able to cap- ture the inherent structure in the given graph and learn good representations for generating the tar- get text. Early work relies on statistical methods or sequence-to-sequence (Seq2Seq) models where input graphs are linearized (Lu et al., 2009; Song et al., 2017; Konstas et al., 2017). Recent studies propose various models based on graph neural net- work (GNN) to encode graphs (Xu et al., 2018b; Beck et al., 2018; Guo et al., 2019; Damonte and Cohen, 2019; Ribeiro et al., 2019). However, these approaches only consider the relations between di- rectly connected nodes, ignore the indirect relations between distance nodes. Inspired by the success of Transformer (Vaswani et al., 2017) which can learn the dependencies between all tokens without regard to their distance, the current state-of-the- art Graph2Seq models (Zhu et al., 2019; Cai and Lam, 2020) are based on Transformer and learn the relations between all nodes no matter they are connected or not. These approaches use shortest relation path between nodes to encode semantic re- lationships. However, they ignore the information of …

Content truncated to 500 words