Maximum Bayes Smatch Ensemble Distillation for AMR Parsing

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5379 - 5392 July 10-15, 2022 ©2022 Association for Computational Linguistics Maximum Bayes Smatch Ensemble Distillation for AMR Parsing Young-Suk Lee†, Ramón Fernandez Astudillo†, Thanh Lam Hoang‡, Tahira Naseem†, Radu Florian†, Salim Roukos† {ysuklee,tanseem,raduf,roukos}@us.ibm.com ramon.astudillo@ibm.com t.l.hoang@ie.ibm.com IBM Research AI†, IBM Research - Ireland‡ Abstract AMR parsing has experienced an unprecen- dented increase in performance in the last three years, due to a mixture of effects including architecture improvements and transfer learn- ing. Self-learning techniques have also played a role in pushing performance forward. How- ever, for most recent high performant parsers, the effect of self-learning and silver data aug- mentation seems to be fading. In this paper we propose to overcome this diminishing returns of silver data by combining Smatch-based en- sembling techniques with ensemble distillation. In an extensive experimental setup, we push single model English parser performance to a new state-of-the-art, 85.9 (AMR2.0) and 84.3 (AMR3.0), and return to substantial gains from silver data augmentation. We also attain a new state-of-the-art for cross-lingual AMR parsing for Chinese, German, Italian and Spanish. Fi- nally we explore the impact of the proposed technique on domain adaptation, and show that it can produce gains rivaling those of human annotated data for QALD-9 and achieve a new state-of-the-art for BioAMR. 1 Introduction Adoption of the Transformer architecture (Vaswani et al., 2017) for Abstract Meaning Representation (AMR) parsing (Cai and Lam, 2020; Fernandez As- tudillo et al., 2020) as well as pretrained language models (Bevilacqua et al., 2021; Zhou et al., 2021b; Bai et al., 2022) have enabled an improvement of above 10 Smatch points (Cai and Knight, 2013), the standard metric, in the last two years. Data augmentation techniques have also shown great success in pushing the state-of-the-art of AMR parsing forward. These include generat- ing silver AMR annotations with a trained parser (Konstas et al., 2017; van Noord and Bos, 2017), with multitask pre-training and fine-tuning (Xu et al., 2020) as well as combining AMR to source text and silver AMR generation (Lee et al., 2020) and stacked pre-training of silver data from dif- ferent models – from low performance to high performance silver data (Xia et al., 2021). How- ever, the latest BART-based state-of-the-art parsers, have shown diminishing returns for data augmenta- tion. Both SPRING (Bevilacqua et al., 2021) and Structured-BART (Zhou et al., 2021b) gain a mere 0.5 Smatch from self-learning, compared with over 1 point gains of the previous, less performant, mod- els. Since performance scores are already above where inter annotator agreement (IAA) is assumed to be, i.e. 83 for newswire and 79 for web text reported in (Banarescu et al., 2013), one possible explanation is that we are reaching some unavoid- able performance plateau. In this work we show that we can achieve sig- nificant performance gains close to 2 Smatch point with the newly proposed data augmentation tech- nique, contrary to the results from the previous …

Content truncated to 500 words