SemEval-2017 Task 9: Abstract Meaning Representation Parsing and Generation

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 536–545, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 9: Abstract Meaning Representation Parsing and Generation Jonathan May and Jay Priyadarshi Information Sciences Institute Computer Science Department University of Southern California jonmay@isi.edu, jpriyada@usc.edu Abstract In this report we summarize the results of the 2017 AMR SemEval shared task. The task consisted of two separate yet related subtasks. In the parsing sub- task, participants were asked to produce Abstract Meaning Representation (AMR) (Banarescu et al., 2013) graphs for a set of English sentences in the biomedical do- main. In the generation subtask, partici- pants were asked to generate English sen- tences given AMR graphs in the news/fo- rum domain. A total of five sites partici- pated in the parsing subtask, and four par- ticipated in the generation subtask. Along with a description of the task and the par- ticipants’ systems, we show various score ablations and some sample outputs. 1 Introduction Abstract Meaning Representation (AMR) is a compact, readable, whole-sentence semantic an- notation (Banarescu et al., 2013). It includes entity identification and typing, PropBank semantic roles (Kingsbury and Palmer, 2002), individual entities playing multiple roles, as well as treatments of modality, negation, etc. AMR abstracts in numer- ous ways, e.g., by assigning the same conceptual structure to fear (v), fear (n), and afraid (adj). Fig- ure 1 gives an example. In 2016 an AMR parsing shared task was held at SemEval (May, 2016). Task participants demon- strated several new directions in AMR parsing technology and also validated the strong perfor- mance of existing parsers. We sought, in 2017, to focus AMR parsing performance on the biomed- ical domain, for which a not insignificant but still relatively small training corpus had been pro- duced. While sentences from this domain are quite (f / fear-01 :polarity "-" :ARG0 ( s / soldier ) :ARG1 ( d / die-01 :ARG1 s )) The soldier was not afraid of dying. The soldier was not afraid to die. The soldier did not fear death. Figure 1: An Abstract Meaning Representation (AMR) with several English renderings. Example borrowed from Pust et al. (2015). formal compared to some of those evaluated in last year’s task, they are also very complex, and have many terms unique to the domain. An example is shown in Figure 2. We continue to use Smatch (Cai and Knight, 2013) as a metric for AMR pars- ing, but we perform additional ablative analysis using the approach proposed by Damonte et al. (2016). Along with parsing into AMR, it is important to encourage improvements in automatic genera- tion of natural language (NL) text from AMR. Hu- mans favor communication in NL. An AI that is able to parse text into AMR at a quality level in- distinguishable from humans may be said to un- derstand NL, but without the ability to render its own semantic representations into NL no human will ever be able to appreciate this. The …

Content truncated to 500 words