XLPT-AMR: Cross-Lingual Pre-Training via Multi-Task Learning for Zero-Shot AMR Parsing and Text Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 896–907 August 1–6, 2021. ©2021 Association for Computational Linguistics 896 XLPT-AMR: Cross-Lingual Pre-Training via Multi-Task Learning for Zero-Shot AMR Parsing and Text Generation Dongqin Xu1 Junhui Li1∗ Muhua Zhu2 Min Zhang1 Guodong Zhou1 1School of Computer Science and Technology, Soochow University, Suzhou, China 2Tencent News, Beijing, China xdqck@live.com, {lijunhui, minzhang, gdzhou}@suda.edu.cn zhumuhua@gmail.com Abstract Due to the scarcity of annotated data, Abstract Meaning Representation (AMR) research is relatively limited and challenging for lan- guages other than English. Upon the avail- ability of English AMR dataset and English-to- X parallel datasets, in this paper we propose a novel cross-lingual pre-training approach via multi-task learning (MTL) for both zero- shot AMR parsing and AMR-to-text genera- tion. Specifically, we consider three types of relevant tasks, including AMR parsing, AMR-to-text generation, and machine transla- tion. We hope that knowledge gained while learning for English AMR parsing and text generation can be transferred to the counter- parts of other languages. With properly pre- trained models, we explore four different fine- tuning methods, i.e., vanilla fine-tuning with a single task, one-for-all MTL fine-tuning, tar- geted MTL fine-tuning, and teacher-student- based MTL fine-tuning. Experimental re- sults on AMR parsing and text generation of multiple non-English languages demonstrate that our approach significantly outperforms a strong baseline of pre-training approach, and greatly advances the state of the art. In detail, on LDC2020T07 we have achieved 70.45%, 71.76%, and 70.80% in Smatch F1 for AMR parsing of German, Spanish, and Italian, re- spectively, while for AMR-to-text generation of the languages, we have obtained 25.69, 31.36, and 28.42 in BLEU respectively. We make our code available on github https:// github.com/xdqkid/XLPT-AMR. 1 Introduction Abstract Meaning Representation (AMR) (Ba- narescu et al., 2013) is a widely used formalism that represents the semantics of a sentence with a directed and acyclic graph. Figure 1 (b) shows an example AMR graph where the nodes such as ∗Corresponding Author: Junhui Li. “doctor” and “give-01” represent concepts, and the edges such as “:ARG0” and “:ARG1” stand for se- mantic relations between two connected concepts. Recent studies on AMR mainly fall in two direc- tions: AMR parsing which converts a sentence into an AMR graph (Flanigan et al., 2014; Wang et al., 2015a; Konstas et al., 2017, to name a few) and its inverse, i.e., AMR-to-text generation that produces a sentence from an AMR graph (Flanigan et al., 2016; Song et al., 2017, 2018, to name a few). Restricted by the availability of annotated cor- pora, most of previous studies on AMR focus on English while very few studies are for Chi- nese and Portuguese (Wang et al., 2018; Sobre- villa Cabezudo et al., 2019; Anchiˆeta and Pardo, 2020). Cross-lingual AMR research, however, has received relatively less attention. In fact, cross- lingual AMR has mainly been studied in the scope of annotation works (Xue et al., 2014; Hajiˇc et al., 2014). Till recently, Damonte and Cohen …

Content truncated to 500 words