RIGA at SemEval-2016 Task 8: Impact of Smatch Extensions and Character-Level Neural Translation on AMR Parsing Accuracy

Proceedings of SemEval-2016, pages 1143–1147, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics RIGA at SemEval-2016 Task 8: Impact of Smatch Extensions and Character-Level Neural Translation on AMR Parsing Accuracy Guntis Barzdins, Didzis Gosko University of Latvia, IMCS and LETA Rainis Blvd. 29, Riga, LV-1459, Latvia guntis.barzdins@lu.lv, didzis.gosko@leta.lv Abstract Two extensions to the AMR smatch scoring script are presented. The first extension com- bines the smatch scoring script with the C6.0 rule-based classifier to produce a human-read- able report on the error patterns frequency ob- served in the scored AMR graphs. This first ex- tension results in 4% gain over the state-of-art CAMR baseline parser by adding to it a manu- ally crafted wrapper fixing the identified CAMR parser errors. The second extension combines a per-sentence smatch with an en- semble method for selecting the best AMR graph among the set of AMR graphs for the same sentence. This second modification auto- matically yields further 0.4% gain when ap- plied to outputs of two nondeterministic AMR parsers: a CAMR+wrapper parser and a novel character-level neural translation AMR parser. For AMR parsing task the character-level neu- ral translation attains surprising 7% gain over the carefully optimized word-level neural translation. Overall, we achieve smatch F1=62% on the SemEval-2016 official scoring set and F1=67% on the LDC2015E86 test set. 1 Introduction Abstract Meaning Representation (AMR) (Bana- rescu et al., 2013) initially was envisioned as an in- termediate representation for semantic machine translation, but has found applications in other NLP fields such as information extraction. 1 Available at http://c60.ailab.lv For SemEval-2016 Task 8 on Meaning Representa- tion Parsing we took a dual approach: besides de- veloping our own neural AMR parser, we also ex- tended the AMR smatch scoring tool (Cai and Knight, 2013) with a rule-based C6.0 classifier1 to guide development of an accuracy-increasing wrap- per for the state-of-art AMR parser CAMR (Wang et al., 2015a; 2015b). A minor gain was also achieved by combining these two approaches in an ensemble. The paper starts with the description of our smatch extensions, followed by the description of our AMR parser and wrapper, and concludes with the results section evaluating the contributions of described techniques to our final SemEval result. 2 Smatch Extensions We describe two extensions2 to the original AMR smatch scoring script. These extensions do not change the smatch algorithm or scores produced, but they extract additional statistical information helpful for improving results of any AMR parser, as will be illustrated in Section 3. 2.1 Visual Smatch with C6.0 Classifier Rules The original AMR smatch scoring metric produces as output only three numbers: precision, recall and F1. When developing an AMR parser, these three numbers alone do not reveal the actual mistakes in the AMR parser output (we call it silver AMR) when compared to the human-annotated gold AMR. 2 Available at https://github.com/didzis/smatchTools 1143 The first step in alleviating this problem is visualiz- ing the mappings produced by the smatch algorithm as part of the scoring process. Figure 1 shows …

Content truncated to 500 words