Align-smatch: A Novel Evaluation Method for Chinese Abstract Meaning Representation Parsing based on Alignment of Concept and Relation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5938–5945 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5938 Align-smatch: A Novel Evaluation Method for Chinese Abstract Meaning Representation Parsing based on Alignment of Concept and Relation Liming Xiao1, Bin Li1, Zhixing Xu1, Kairui Huo1, Minxuan Feng1, Junsheng Zhou2 and Weiguang Qu2 1School of Chinese Language and Literature, Nanjing Normal University, Nanjing, Jiangsu, China 2School of Computer, Electronics and Information, Nanjing Normal University, Nanjing, Jiangsu, China lmxiao1@foxmail.com, libin.njnu@gmail.com, xzx0828@live.com, kairui.huo.nj@gmail.com, fennel 2006@163.com, {zhoujs, wgqu}@njnu.edu.cn Abstract Abstract Meaning Representation is a sentence-level meaning representation, which abstracts the meaning of sentences into a rooted acyclic directed graph. With the continuous expansion of Chinese AMR corpus, more and more scholars have developed parsing systems to automatically parse sentences into Chinese AMR. However, the current parsers can’t deal with concept alignment and relation alignment, let alone the evaluation methods for AMR parsing. Therefore, to make up for the vacancy of Chinese AMR parsing evaluation methods, based on AMR evaluation metric smatch, we have improved the algorithm of generating triples so that to make it compatible with concept alignment and relation alignment. Finally, we obtain a new integrity metric Align-smatch for parsing evaluation. A comparative research then was conducted on 20 manually annotated AMR and gold AMR, with the result that Align-smatch works well in alignments and more robust in evaluating arcs. We also put forward some fine-grained metric for evaluating concept alignment, relation alignment and implicit concepts, in order to further measure parsers’ performance in subtasks. Keywords: Abstract Meaning Representation, Align-smatch, Concept Alignment, Relation Alignment 1. Introduction With the growing maturity of morphological analy- sis and syntactic analysis techniques, natural language processing in general has advanced to the level of semantic analysis. As the crux part, sentence-level meaning has occupied the core position of semantic analysis research (Sun et al., 2014). To address the lack of whole-sentence semantic representation and the domain-dependent problem of sentence seman- tic annotation, Banarescu et al. proposed a domain- independent whole-sentence semantic representation method in 2013, which is Abstract Meaning Represen- tation (AMR) that can abstract the meaning of a sen- tence with a single-rooted, acyclic and directed graph (Banarescu et al., 2013). AMR not only depicts the phenomenon of argument sharing formed by a noun governed by multiple predicates but also allows to sup- plement the latent semantic to represent the sentence meaning completely. The great capability of semantic representation renders AMR the widespread attention upon its introduction, and countless articles on vari- ous aspects such as AMR automatic parsing and AMR transformation applications have emerged as well. Li et al. (2016) introduced AMR into Chinese and made corresponding adjustments according to the lin- guistic characteristics of Chinese such as adding la- bels representing semantic relations like “aspect” and “quantifier”, specifying the treatment of unique struc- tures like clutch words and most importantly, retaining function words as nodes or labeling them on directed arcs. And in particular, they also …

Content truncated to 500 words