GPT-too: A Language-Model-First Approach for AMR-to-Text Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1846–1852 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1846 GPT-too: A Language-Model-First Approach for AMR-to-Text Generation Manuel Mager1∗ Ram´on Fernandez Astudillo2 Tahira Naseem2 Md Arafat Sultan2 Young-Suk Lee2 Radu Florian2 Salim Roukos2 1 Institute for Natural Language Processing, University of Stuttgart, Germany 2 IBM Research AI, Yorktown Heights, NY 10598, USA manuel.mager@ims.uni-stuttgart.de {ramon.astudillo,arafat.sultan}@ibm.com {tnaseem, ysuklee}@us.ibm.com Abstract Abstract Meaning Representations (AMRs) are broad-coverage sentence-level semantic graphs. Existing approaches to generating text from AMR have focused on training sequence- to-sequence or graph-to-sequence models on AMR annotated data only. In this paper, we propose an alternative approach that combines a strong pre-trained language model with cy- cle consistency-based re-scoring. Despite the simplicity of the approach, our experimental results show these models outperform all pre- vious techniques on the English LDC2017T10 dataset, including the recent use of transformer architectures. In addition to the standard eval- uation metrics, we provide human evalua- tion experiments that further substantiate the strength of our approach. 1 Introduction Abstract Meaning Representation (AMR) (Ba- narescu et al., 2013) is a rooted, directed, acyclic graph with labeled edges (relations) and nodes (concepts) expressing “who is doing what to whom”. AMR-to-text generates sentences repre- senting the semantics underlying an AMR graph. ersInitial works in AMR-to-text used transduc- (Flanigan et al., 2016), phrase-based ma- chine translation (Pourdamghani et al., 2016) and neural sequence-to-sequence (seq2seq) models with linearized graphs (Konstas et al., 2017). Cao and Clark (2019) leverage constituency parsing for generation. Beck et al. (2018) improve upon prior RNN graph encoding (Song et al., 2018) with Levi Graph Transformations. Damonte and Co- hen (2019) compare multiple representations and find graph encoders to be the best. Guo et al. (2019) use RNN graph encoders with dense graph convolutional encoding. Ribeiro et al. (2019) ∗ This research was done during an internship at IBM Research AI. use RNN encoders with dual graph representa- tions. Transformer-based seq2seq (Vaswani et al., 2017) was first applied to AMR-to-text in (Sinh and Le Minh, 2019). Zhu et al. (2019) greatly improve over the prior state-of-the-art by modifying self-attention to account for AMR graph structure. Using transformers has also been recently explored by Wang et al. (2020) who pro- pose a mutli-head graph attention mechanism. Pre-trained transformer representations (Rad- ford et al., 2018; Devlin et al., 2019; Radford et al., 2019) use transfer learning to yield pow- erful language models that considerably outper- form the prior art. They have also shown great success when fine-tuned to particular text gener- ation tasks (See et al., 2019; Zhang et al., 2019; Keskar et al., 2019). Given their success, it would be desirable to apply pre-trained transformer mod- els to a graph-to-text task like AMR-to-text, but the need for graph encoding precludes in princi- ple that option. Feeding the network with some sequential representation of the graph, such as a topological sorting, looses some of the graphs rep- resentational power. Complex graph annotations, such as AMR, also …

Content truncated to 500 words