Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6279–6290 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6279 Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing Jiawei Zhoum Tahira Naseem3 Ramón Fernandez Astudillo3 Young-Suk Lee3 Radu Florian3 Salim Roukos3 mHarvard University 3IBM Research mjzhou02@g.harvard.edu 3ramon.astudillo@ibm.com 3{tnaseem, ysuklee, raduf, roukos}@us.ibm.com Abstract Predicting linearized Abstract Meaning Rep- resentation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the de- sirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization. 1 Introduction The task of Abstract Meaning Representation (AMR) parsing translates a natural sentence into a rooted directed acyclic graph capturing the se- mantics of the sentence, with nodes representing concepts and edges representing their relations (Ba- narescu et al., 2013). Recent works utilizing pre- trained encoder-decoder language models show great improvements in AMR parsing results (Xu et al., 2020; Bevilacqua et al., 2021). These ap- proaches avoid explicit modeling of the graph struc- ture. Instead, they directly predict the linearized AMR graph treated as free text. While the use of pre-trained Transformer encoders is widely ex- tended in AMR parsing, the use of pre-trained Transformer decoders is recent and has shown to be very effective, maintaining current state-of-the-art results (Bevilacqua et al., 2021). These approaches however lack certain desirable properties. There are no structural guarantees of graph well-formedness, i.e. the model may predict strings that can not be decoded into valid graphs, and post-processing is required. Furthermore, pre- dicting AMR linearizations ignores the implicit alignments between graph nodes and words, which provide a strong inductive bias and are useful for downstream AMR applications (Mitra and Baral, 2016; Liu et al., 2018; Vlachos et al., 2018; Kapa- nipathi et al., 2021; Naseem et al., 2021). On the other hand, transition-based AMR parsers (Wang et al., 2015; Ballesteros and Al-Onaizan, 2017a; Astudillo et al., 2020; Zhou et al., 2021) operate over the tokens of the input sentence, gen- erating the graph incrementally. They implicitly model graph structural constraints through transi- tions and yield alignments by construction, thus guaranteeing graph well-formedness.1 However, it remains unclear whether explicit modeling of structure is still beneficial for AMR parsing in the presence of powerful pre-trained language models and their strong free text generation abilities. In …

Content truncated to 500 words