Modeling Graph Structure in Transformer for Better AMR-to-Text Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5459–5468, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5459 Modeling Graph Structure in Transformer for Better AMR-to-Text Generation Jie Zhu1 Junhui Li1∗ Muhua Zhu2 Longhua Qian1 Min Zhang1 Guodong Zhou1 1School of Computer Science and Technology, Soochow University, Suzhou, China 2Alibaba Group, Hangzhou, China zhujie951121@gmail.com, {lijunhui, qianlonghua, minzhang, gdzhou}@suda.edu.cn muhua.zmh@alibaba-inc.com Abstract Recent studies on AMR-to-text generation often formalize the task as a sequence-to- sequence (seq2seq) learning problem by con- verting an Abstract Meaning Representation (AMR) graph into a word sequence. Graph structures are further modeled into the seq2seq framework in order to utilize the structural in- formation in the AMR graphs. However, pre- vious approaches only consider the relations between directly connected concepts while ig- noring the rich structure in AMR graphs. In this paper we eliminate such a strong limita- tion and propose a novel structure-aware self- attention approach to better modeling the re- lations between indirectly connected concepts in the state-of-the-art seq2seq model, i.e., the Transformer. In particular, a few different methods are explored to learn structural rep- resentations between two concepts. Experi- mental results on English AMR benchmark datasets show that our approach significantly outperforms the state of the art with 29.66 and 31.82 BLEU scores on LDC2015E86 and LDC2017T10, respectively. To the best of our knowledge, these are the best results achieved so far by supervised models on the bench- marks. 1 Introduction AMR-to-text generation is a task of automatically generating a natural language sentence from an Abstract Meaning Representation (AMR) graph. Due to the importance of AMR as a widely adopted semantic formalism in representing the meaning of a sentence (Banarescu et al., 2013), AMR has become popular in semantic representa- tion and AMR-to-text generation has been draw- ing more and more attention in the last decade. As the example in Figure 1(a) shows, nodes, such as he and convict-01, represent semantic concepts ∗Corresponding Author: Junhui Li. and edges, such as “:ARG1” and “:quant”, re- fer to semantic relations between the concepts. Since two concepts close in an AMR graph may map into two segments that are distant in the cor- responding sentence, AMR-to-text generation is challenging. For example in Figure 1, the neigh- boring concepts he and convict-01 correspond to the words he and convicted which locate at the dif- ferent ends of the sentence. To address the above mentioned challenge, re- cent studies on AMR-to-text generation regard the task as a sequence-to-sequence (seq2seq) learning problem by properly linearizing an AMR graph into a sequence (Konstas et al., 2017). Such an input representation, however, is apt to lose useful structural information due to the removal of reen- trant structures for linearization. To better model graph structures, previous studies propose various graph-based seq2seq models to incorporate graphs as an additional input representation (Song et al., 2018; Beck et al., 2018; Damonte and Cohen, 2019). Although such graph-to-sequence models …

Content truncated to 500 words