Automatic Accuracy Prediction for AMR Parsing

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 212–223 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 212 Automatic Accuracy Prediction for AMR Parsing Juri Opitz and Anette Frank Research Training Group AIPHES Leibniz ScienceCampus “Empirical Linguistics and Computational Language Modeling” Department for Computational Linguistics 69120 Heidelberg {opitz,frank}@cl.uni-heidelberg.de Abstract Abstract Meaning Representation (AMR) re- presents sentences as directed, acyclic and rooted graphs, aiming at capturing their mean- ing in a machine readable format. AMR pars- ing converts natural language sentences into such graphs. However, evaluating a parser on new data by means of comparison to manually created AMR graphs is very costly. Also, we would like to be able to detect parses of ques- tionable quality, or preferring results of alter- native systems by selecting the ones for which we can assess good quality. We propose AMR accuracy prediction as the task of predicting several metrics of correctness for an automati- cally generated AMR parse – in absence of the corresponding gold parse. We develop a neural end-to-end multi-output regression model and perform three case studies: firstly, we evalu- ate the model’s capacity of predicting AMR parse accuracies and test whether it can reli- ably assign high scores to gold parses. Sec- ondly, we perform parse selection based on predicted parse accuracies of candidate parses from alternative systems, with the aim of im- proving overall results. Finally, we predict system ranks for submissions from two AMR shared tasks on the basis of their predicted parse accuracy averages. All experiments are carried out across two different domains and show that our method is effective. 1 Introduction Abstract Meaning Representation (AMR) (Ba- narescu et al., 2013) represents the semantic struc- ture of a sentence, including concepts, seman- tic operators and relations, sense-disambiguated predicates and their arguments. As a machine readable representation of the meaning of a sen- tence, AMR is potentially useful for many NLP tasks. Among other applications it has been used in machine translation (Jones et al., 2012), text (a / asbestos :polarity - :time (n / now) :location (t / thing :ARG1-of (p / produce-01 :ARG0 (w / we)))) Figure 1: Humanly produced AMR for: There is no as- bestos in our products now. Numbered predicates refer to PropBank senses (Palmer et al., 2005). summarization (Liu et al., 2015; Dohare and Kar- nick, 2017) and question answering (Mitra and Baral, 2016). Since the introduction of AMR, many approaches to AMR parsing have been pro- posed: graph-based pipeline systems which rely on an alignment step (Flanigan et al., 2014, 2016) or transition-based parsers relying on dependency annotation (Wang et al., 2015b,a, 2016a). In the following we will denote the former by JAMR and the latter by CAMR. More recently, end- to-end neural systems have been proposed which produce linearized AMR graphs within character- based (van Noord and Bos, 2017b) or word-based (Konstas et al., 2017) encoding models. Both ap- proaches greatly profit from large amounts of sil- ver training data. The silver data is …

Content truncated to 500 words