SemEval-2016 Task 8: Meaning Representation Parsing

Proceedings of SemEval-2016, pages 1063–1073, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 8: Meaning Representation Parsing Jonathan May Information Sciences Institute University of Southern California jonmay@isi.edu Abstract In this report we summarize the results of the SemEval 2016 Task 8: Meaning Representa- tion Parsing. Participants were asked to gener- ate Abstract Meaning Representation (AMR) (Banarescu et al., 2013) graphs for a set of English sentences in the news and discussion forum domains. Eleven sites submitted valid systems. The availability of state-of-the-art baseline systems was a key factor in lower- ing the bar to entry; many submissions relied on CAMR (Wang et al., 2015b; Wang et al., 2015a) as a baseline system and added exten- sions to it to improve scores. The evaluation set was quite difficult to parse, particularly due to creative approaches to word representation in the web forum portion. The top scoring sys- tems scored 0.62 F1 according to the Smatch (Cai and Knight, 2013) evaluation heuristic. We show some sample sentences along with a comparison of system parses and perform quantitative ablative studies. 1 Introduction Abstract Meaning Representation (AMR) is a com- pact, readable, whole-sentence semantic annota- tion (Banarescu et al., 2013). It includes entity identification and typing, PropBank semantic roles (Kingsbury and Palmer, 2002), individual entities playing multiple roles, as well as treatments of modality, negation, etc. AMR abstracts in numerous ways, e.g., by assigning the same conceptual struc- ture to fear (v), fear (n), and afraid (adj). Figure 1 gives an example. With the recent public release of a sizeable cor- pus of English/AMR pairs (LDC2014T12), there has (f / fear-01 :polarity "-" :ARG0 ( s / soldier ) :ARG1 ( d / die-01 :ARG1 s )) The soldier was not afraid of dying. The soldier was not afraid to die. The soldier did not fear death. Figure 1: An Abstract Meaning Representation (AMR) with several English renderings. Example borrowed from Pust et al. (2015). been substantial interest in creating parsers to re- cover this formalism from plain text. Several parsers were released in the past couple of years (Flanigan et al., 2014; Wang et al., 2015b; Werling et al., 2015; Wang et al., 2015a; Artzi et al., 2015; Pust et al., 2015). This body of work constitutes many diverse and interesting scientific contributions, but it is diffi- cult to adequately determine which parser is numer- ically superior, due to heterogeneous evaluation de- cisions and the lack of a controlled blind evaluation. The purpose of this task, therefore, was to provide a competitive environment in which to determine one winner and award a trophy to said winner. 2 Training Data LDC released a new corpus of AMRs (LDC2015E86), created as part of the DARPA DEFT program, in August of 2015. The new cor- pus, which was annotated by teams at SDL, LDC, and the University of Colorado, and supervised by Ulf Hermjakob at USC/ISI, is an extension of pre- 1063 vious releases (LDC2014E41 and LDC2014T12). It contains 19,572 sentences …

Content truncated to 500 words