Accounting for Language Effect in the Evaluation of Cross-lingual AMR Parsers

Proceedings of the 29th International Conference on Computational Linguistics, pages 3824–3834 October 12–17, 2022. 3824 Accounting for Language Effect in the Evaluation of Cross-lingual AMR Parsers Shira Wein Georgetown University sw1158@georgetown.edu Nathan Schneider Georgetown University nathan.schneider@georgetown.edu Abstract Cross-lingual Abstract Meaning Representa- tion (AMR) parsers are currently evaluated in comparison to gold English AMRs, despite parsing a language other than English, due to the lack of multilingual AMR evaluation met- rics. This evaluation practice is problematic because of the established effect of source lan- guage on AMR structure. In this work, we present three multilingual adaptations of mono- lingual AMR evaluation metrics and compare the performance of these metrics to sentence- level human judgments. We then use our most highly correlated metric to evaluate the output of state-of-the-art cross-lingual AMR parsers, finding that Smatch may still be a use- ful metric in comparison to gold English AMRs, while our multilingual adaptation of S2match (XS2match) is best for comparison with gold in-language AMRs. 1 Introduction The Abstract Meaning Representation (AMR; Ba- narescu et al., 2013) formalism captures the mean- ing of a sentence or phrase as a rooted, directed acyclic graph. Nodes correspond to concepts and the labeled edges reflect the relations between con- cepts. For example, the annotation in Figure 1 features the AMR annotation of the sentence “we will try not to make a mistake” in both PENMAN (text-based) and graph form. The edge labels can be arguments (core roles, denoted as :argN), or one of a number of non-core roles such as :location or :manner. Cross-lingual AMR parsers convert non-English text to (English-focused) AMR graphs. As there are no existing multilingual AMR evaluation metrics and due to the limited availability of non-English gold AMR annotations, these cross-lingual AMR parsers have only ever been evaluated in compar- ison to gold English AMRs. This established ap- proach of comparison to English AMRs using the monolingual Smatch metric needs to be considered (v2 / try-01 :ARG0 (v1 / we) :ARG1 (v3 / mistake-02 :polarity - :ARG0 v1)) Figure 1: An AMR annotation for the sentence “We will try not to make a mistake,” in PENMAN text-based notation and as a rooted graph. The diagram was made on the AMREager website (Damonte et al., 2017). more carefully because previous work has estab- lished that the source language has a dramatic effect on the cross-lingual AMRs (Damonte, 2019; Wein and Schneider, 2021, 2022). We argue that cross-lingual AMR parsing should represent the semantics, beyond the lexicon, faith- fully to the source language. Enabled by work developing sizable Spanish (Wein et al., 2022) and Chinese (Li et al., 2016) gold AMR corpora, we propose that cross-lingual AMR parsers should be evaluated on gold AMRs that match the lan- guage being parsed. In this work, we adapt three monolingual AMR evaluation metrics to a mul- tilingual setting, and evaluate the performance of these metrics in comparison to human judgments of cross-lingual sentence similarity. We show that our adaptation of S2match (Opitz et al., 2020) which leverages …

Content truncated to 500 words