A Human Evaluation of AMR-to-English Generation Systems

Proceedings of the 28th International Conference on Computational Linguistics, pages 4773–4786 Barcelona, Spain (Online), December 8-13, 2020 4773 A Human Evaluation of AMR-to-English Generation Systems Emma Manning Shira Wein Nathan Schneider Georgetown University {esm76, sw1158, nathan.schneider}@georgetown.edu Abstract Most current state-of-the art systems for generating English text from Abstract Meaning Repre- sentation (AMR) have been evaluated only using automated metrics, such as BLEU, which are known to be problematic for natural language generation. In this work, we present the results of a new human evaluation which collects fluency and adequacy scores, as well as categorization of error types, for several recent AMR generation systems. We discuss the relative quality of these systems and how our results compare to those of automatic metrics, finding that while the metrics are mostly successful in ranking systems overall, collecting human judgments allows for more nuanced comparisons. We also analyze common errors made by these systems. 1 Introduction Abstract Meaning Representation, or AMR (Banarescu et al., 2013), is a representation of the meaning of a sentence as a rooted, labeled, directed acyclic graph. For example, (l / label-01 :ARG0 (c / country :wiki "Georgia_(country)" :name (n / name :op1 "Georgia")) :ARG1 (s / support-01 :ARG0 (c2 / country :wiki "Russia" :name (n2 / name :op1 "Russia"))) :ARG2 (a / act-02 :mod (a2 / annex-01))) represents the sentence “Georgia labeled Russia’s support an act of annexation.” AMR does not represent some morphological and syntactic details such as tense, number, definiteness, and word order; thus, this same AMR could also represent alternate phrasings such as “Russia’s support is being labeled an act of annexation by Georgia.” AMR generation is the task of generating a sentence in natural language (in this case, English) from an AMR graph. This has applications to a range of NLP tasks, including summarization (Liao et al., 2018) and machine translation (Song et al., 2019). Like other Natural Language Generation (NLG) tasks, this is difficult to evaluate due to the range of possible valid sentences corresponding to any single AMR. Currently, AMR generation systems are often evaluated only with automatic metrics that compare a generated sentence to a single human-authored reference; for AMR, this is the sentence from which the AMR graph was created. However, there is evidence that these metrics may not be a good representation of human judgments for AMR generation (May and Priyadarshi, 2017) and NLG in general (see §2.1). Thus, in this work, we present a new human evaluation of several recent AMR generation systems, most of which had not previously been manually evaluated. Our methodology (§3) differs in several ways from previous evaluations of AMR generation, including separate direct assessment of fluency and adequacy; and asking annotators to evaluate sentences without comparison to a reference, in order to avoid biasing them toward a particular wording. We analyze (§4) what our results show about the relative quality of the systems and how this compares to their scores from automatic metrics, finding that these metrics This work is licensed under a Creative …

Content truncated to 500 words