Improving AMR Parsing with Sequence-to-Sequence Pre-training
30692Authors
nan
Abstract
AbstractIn the literature, the research on abstract meaning representation (AMR) parsing is much restricted by the size of human-curated dataset which is critical to build an AMR parser with good performance. To alleviate such data size restriction, pre-trained models have been drawing more and more attention in AMR parsing. However, previous pre-trained models, like BERT, are implemented for general purpose which may not work as expected for the specific task of AMR parsing. In this paper, we focus on sequence-to-sequence (seq2seq) AMR parsing and propose a seq2seq pre-training approach to build pre-trained models in both single and joint way on three relevant tasks, i.e., machine translation, syntactic parsing, and AMR parsing itself. Moreover, we extend the vanilla fine-tuning method to a multi-task learning fine-tuning method that optimizes for the performance of AMR parsing while endeavors to preserve the response of pre-trained models. Extensive experimental results on two English benchmark datasets show that both the single and joint pre-trained models significantly improve the performance (e.g., from 71.5 to 80.2 on AMR 2.0), which reaches the state of the art. The result is very encouraging since we achieve this with seq2seq models rather than complex models. We make our code and model available at https://github.com/xdqkid/S2S-AMR-Parser.
Keywords
Full Content
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2501–2511, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2501 Improving AMR Parsing with Sequence-to-Sequence Pre-training Dongqin Xu1 Junhui Li1∗ Muhua Zhu2 Min Zhang1 Guodong Zhou1 1School of Computer Science and Technology, Soochow University, Suzhou, China 2Tencent News, Beijing, China xdqck@live.com, {lijunhui, minzhang, gdzhou}@suda.edu.cn muhuazhu@tencent.com Abstract In the literature, the research on abstract mean- ing representation (AMR) parsing is much re- stricted by the size of human-curated dataset which is critical to build an AMR parser with good performance. To alleviate such data size restriction, pre-trained models have been draw- ing more and more attention in AMR parsing. However, previous pre-trained models, like BERT, are implemented for general purpose which may not work as expected for the spe- cific task of AMR parsing. In this paper, we fo- cus on sequence-to-sequence (seq2seq) AMR parsing and propose a seq2seq pre-training ap- proach to build pre-trained models in both sin- gle and joint way on three relevant tasks, i.e., machine translation, syntactic parsing, and AMR parsing itself. Moreover, we extend the vanilla fine-tuning method to a multi-task learning fine-tuning method that optimizes for the performance of AMR parsing while en- deavors to preserve the response of pre-trained models. Extensive experimental results on two English benchmark datasets show that both the single and joint pre-trained models signif- icantly improve the performance (e.g., from 71.5 to 80.2 on AMR 2.0), which reaches the state of the art. The result is very encourag- ing since we achieve this with seq2seq mod- els rather than complex models. We make our code and model available at https:// github.com/xdqkid/S2S-AMR-Parser. 1 Introduction Abstract meaning representation (AMR) parsing aims to translate a textual sentence into a directed and acyclic graph which consists of concept nodes and edges representing the semantic relations be- tween the nodes (Banarescu et al., 2013). Previ- ous studies focus on building diverse approaches to modeling the structure in AMR graphs, such as tree- based approaches (Wang et al., 2015b; Groschwitz ∗Corresponding Author: Junhui Li. (a) Input China considers Germany the most important trade partner of Europe. (c / consider-01 :ARG0 (c2 / country :wiki "China" :name (n / name :op1 "China")) :ARG1 (p / partner-01 :ARG1 (c4 / country :wiki "Germany" :name (n3 / name :op1 "Germany")) :mod (i / important :degree (m / most)) :mod (t / trade-01) :location (c3 / continent :wiki "Europe" :name (n2 / name :op1 "Europe")))) ( consider-01 : ARG0 ( country : name ( name : op1 " China " ) ) : ARG1 ( partner-01 : ARG1 ( country : name ( name : op1 " Germany " ) ) : mod ( important : degree ( most ) ) : mod ( trade-01 ) : location ( continent : name ( name : op1 " Europe " ) ) ) ) (b) AMR (c) AMR Linearization Figure 1: An example of seq2seq-based AMR parsing. et al., 2018), graph-based approaches (Flanigan et al., 2014; Werling et …
Content truncated to 500 words
Labeled Text
AbstractIn the literature, the research on abstract meaning representation (AMR) parsing is much restricted by the size of human-curated dataset which is critical to build an AMR parser with good performance. To alleviate such data size restriction, pre-trained models have been drawing more and more attention in AMR parsing. However, previous pre-trained models, like BERT, are implemented for general purpose which may not work as expected for the specific task of AMR parsing. In this paper, we focus on sequence-to-sequence (seq2seq) AMR parsing and propose a seq2seq pre-training approach to build pre-trained models in both single and joint way on three relevant tasks, i.e., machine translation, syntactic parsing, and AMR parsing itself. Moreover, we extend the vanilla fine-tuning method to a multi-task learning fine-tuning method that optimizes for the performance of AMR parsing while endeavors to preserve the response of pre-trained models. Extensive experimental results on two English benchmark datasets show that both the single and joint pre-trained models significantly improve the performance (e.g., from 71.5 to 80.2 on AMR 2.0), which reaches the state of the art. The result is very encouraging since we achieve this with seq2seq models rather than complex models. We make our code and model available at
Publication Details
- Paper ID:
- 30692
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan