SemBleu: A Robust Metric for AMR Parsing Evaluation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4547–4552 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4547 SEMBLEU: A Robust Metric for AMR Parsing Evaluation Linfeng Song and Daniel Gildea Department of Computer Science University of Rochester Rochester, NY 14627 Abstract Evaluating AMR parsing accuracy involves comparing pairs of AMR graphs. The major evaluation metric, SMATCH (Cai and Knight, 2013), searches for one-to-one mappings be- tween the nodes of two AMRs with a greedy hill-climbing algorithm, which leads to search errors. We propose SEMBLEU, a robust met- ric that extends BLEU (Papineni et al., 2002) to AMRs. It does not suffer from search er- rors and considers non-local correspondences in addition to local ones. SEMBLEU is fully content-driven and punishes situations where a system’s output does not preserve most in- formation from the input. Preliminary experi- ments on both sentence and corpus levels show that SEMBLEU has slightly higher consistency with human judgments than SMATCH. Our code is available at http://github.com/ freesunshine0316/sembleu. 1 Introduction Abstract Meaning Representation (AMR) (Ba- narescu et al., 2013) is a semantic formalism where the meaning of a sentence is encoded as a rooted, directed graph. Figure 1 shows two AMR graphs in which the nodes (such as “girl” and “leave-11”) represent AMR concepts and the edges (such as “ARG0” and “ARG1”) represent re- lations between the concepts. The task of parsing sentences into AMRs has received increasing at- tention, due to the demand for better natural lan- guage understanding. Despite the large amount of work on AMR pars- ing (Flanigan et al., 2014; Artzi et al., 2015; Pust et al., 2015; Peng et al., 2015; Buys and Blunsom, 2017; Konstas et al., 2017; Wang and Xue, 2017; Ballesteros and Al-Onaizan, 2017; Lyu and Titov, 2018; Peng et al., 2018; Groschwitz et al., 2018; Guo and Lu, 2018), little attention has been paid to evaluating the parsing results, leaving SMATCH :ARG0 ask-01 girl boy :ARG0 leave-11 :ARG1 :ARG0 make-01 woman 2 :quant pie :ARG1 (a) (b) Figure 1: Two AMR examples meaning “The girl asked the boy to leave.” and “The woman made two pies.”, respectively. Their SMATCH score is 25 (%). Figure 2: Average, minimal and maximal SMATCH scores over 100 runs on 100 sentences. The running time increases from 6.6 seconds (r=1) to 21.0 (r = 4). (Cai and Knight, 2013) as the only overall perfor- mance metric. Damonte et al. (2017) developed a suite of fine-grained performance measures based on the node mappings of SMATCH (see below). SMATCH suffers from two major drawbacks: first, it is based on greedy hill-climbing to find a one-to-one node mapping between two AMRs (finding the exact best mapping is NP-complete). The search errors weaken its robustness as a met- ric. To enhance robustness, the hill-climbing search is executed multiple times with random restarts. This decreases efficiency and, more im- portantly, does not eliminate search errors. Fig- ure 2 shows the means and error bounds of SMATCH …

Content truncated to 500 words